Test for documented examples (#590)

* Tests for examples

The examples now have test files to assert that RabbitMQ has been
configured as intended by the example. Some examples do not have tests
because they are either very basic or its use case is sufficiently
tested in code.

docs/examples/README.md

@@ -0,0 +1,39 @@
+## RabbitMQ Cluster Operator examples
+
+This section contains examples on how to configure some features of RabbitMQ.
+The examples are common use cases e.g. [tls](./tls) to configure specific RabbitMQ features
+or how RabbitMQ Pods will behave inside Kubernetes.
+
+### Testing framework
+
+Some examples have tests to ensure that the example has achieved its intention. Any new examples
+should have tests. Exceptions apply if the feature itself is sufficiently tested in the code, for
+example, [resource limits](./resource-limits) is tested in the code to ensure that given a set of
+inputs, the Pod resource requests are configured accordingly. Duplicating the same assertion here
+would not make sense, and `exec`'ing into the container to ensure that Kubernetes has respected
+the resource requests would fall under Kubernetes Core testing.
+
+### Writing tests for examples
+
+Every folder with an example must have a file `test.sh`. This executable Bash file has to assert on
+the state of RabbitMQ to ensure that it has been configured according to the expectations of the example.
+
+If the test requires some preparation or setup, a file `setup.sh` can be provided to be executed
+**before** the RabbitMQ cluster is created. This file must be a Bash executable. This file can assume
+that `kubectl` is configured to execute commands against a working Kubernetes cluster.
+
+The script `test.sh` will be executed **after** `AllReplicasReady` condition is `True` in `RabbitmqCluster`
+object. The script `test.sh` should exit with code 0 if all assertions were successful; the script `test.sh` should
+exit with non-zero code if any test or assertion failed. The same is expected for `setup.sh`.
+
+If the example should not run any tests because of the reasons mentioned above, the folder should contain
+a file `.ci-skip`, so that the example is not considered in our tests.
+
+Once the `test.sh` script has completed, the namespace where the example was applied will be deleted. This allows
+for a clean slate for the next test to execute. This also means that `test.sh` does not need to
+tear down namespaced resources.
+
+The test and setup scripts can assume that [Cert Manager](https://cert-manager.io/) is installed and available.
+There is also a cluster issuer to produce self-signed certificates, named `selfsigned-issuer`. It is also
+acceptable to create local `Issuer`s when needed.
+

docs/examples/additionalPorts/test.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+set -x
+port12345=$(kubectl get pod -l app.kubernetes.io/name=rabbit \
+  -ojsonpath='{.items[0].spec.containers[0].ports[?(@.containerPort==12345)].name}' 2> /dev/null)
+## kubectl std. error is redirectd to null because the error output of jsonpath
+## is not very helpful to troubleshoot
+
+[[ "$port12345" == "additional-port" ]] || exit 1
+

docs/examples/community-plugins/test.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+kubectl exec community-plugins-server-0 -- rabbitmq-plugins is_enabled rabbitmq_message_timestamp
+

docs/examples/federation-over-tls/README.md

@@ -1,15 +1,29 @@
 # Federation Over TLS Example
 
-This is the a more complex example of deploying two `RabbitmqCluster`s and setting up federation between them. Upstream cluster has TLS enabled and therefore federation works over a TLS connection.
+This is a more complex example of deploying a `RabbitmqCluster` and setting up federation between two virtual hosts. The
+cluster has TLS enabled and therefore federation works over a TLS connection.
 
-First, please follow [TLS example](../tls) to create a TLS secret. Once you have a secret, run the `setup.sh` script:
+First, please follow [TLS example](../tls) to create a TLS secret. Alternatively, if you have
+[cert-manager](https://cert-manager.io/docs/installation/kubernetes/), you can apply the certificate `certificate.yaml` file.
+The certificate expects to have a `ClusterIssuer` named `selfsigned-issuer`. Feel free to adapt this value accordingly to your
+cert-manager installation.
 
-```shell
-./setup.sh
+In addition, you have to create a ConfigMap to import the definitions with the topology pre-defined.
+
+```bash
+kubectl apply -f certificate.yaml
+kubectl create configmap definitions --from-file=./definitions.json
 ```
 
-The script will stop at some point and ask you to run `sudo kubefwd svc`. This is so that `rabbitmqadmin` can connect to the Management API and configure federation.
+The example has two vhosts "upstream" and "downstream". Both vhosts have a fanout exchange 'example', bound to quorum queue 'qq2'
+in "upstream", quorum queue 'qq1' and classic queue 'cq1' in "downstream". There is a policy in the "downstream" to federate
+the exchange 'example'. All messages published to 'example' exchange in "upstream" will be federated/copied to 'example' exchange
+in "downstream", where the bindings will be applied.
+
+The definitions also import two users: `admin` and `federation`, with passwords matching the usernames (e.g. admin/admin). Note that
+due to the imported definitions, the credentials created by the Operator in Secret `federation-default-user` won't be applied/effective.
 
-Therefore to use this script as-is, you need both [kubefwd](https://github.com/txn2/kubefwd) and [rabbitmqadmin](https://www.rabbitmq.com/management-cli.html) CLIs on your machine.
+If you don't want to import the definitions, or want to manually create the topology, the file `rabbitmq-without-import.yaml` will
+create a RabbitMQ single-node, with federation plugins enabled and TLS configured.
 
 Learn [more about RabbitMQ Federation](https://www.rabbitmq.com/federation.html).

docs/examples/federation-over-tls/certificate.yaml

@@ -0,0 +1,11 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: selfsigned-cert
+spec:
+  dnsNames:
+    - "federation-server-0.federation-nodes.default"
+  secretName: tls-secret
+  issuerRef:
+    kind: ClusterIssuer
+    name: selfsigned-issuer

docs/examples/federation-over-tls/definitions.json

@@ -0,0 +1 @@
+{"rabbit_version":"3.8.9","rabbitmq_version":"3.8.9","product_name":"RabbitMQ","product_version":"3.8.9","users":[{"name":"federation","password_hash":"m3R0yEjfO0fv/Ye/oMn//10Uq5F3j49Ro2P8HE0z27DSJ9iT","hashing_algorithm":"rabbit_password_hashing_sha256","tags":""},{"name":"admin","password_hash":"0l9GJ5sSPU5FGnf95CKOyT/+/CLK+My13+G9fTgK0tn5EyDH","hashing_algorithm":"rabbit_password_hashing_sha256","tags":"administrator"}],"vhosts":[{"name":"/"},{"name":"downstream"},{"name":"upstream"}],"permissions":[{"user":"admin","vhost":"/","configure":".*","write":".*","read":".*"},{"user":"admin","vhost":"downstream","configure":".*","write":".*","read":".*"},{"user":"federation","vhost":"downstream","configure":".*","write":".*","read":".*"},{"user":"admin","vhost":"upstream","configure":".*","write":".*","read":".*"},{"user":"federation","vhost":"upstream","configure":".*","write":".*","read":".*"}],"topic_permissions":[],"parameters":[{"value":{"ack-mode":"on-confirm","trust-user-id":false,"uri":"amqps://federation:[email protected]/upstream"},"vhost":"downstream","component":"federation-upstream","name":"example"},{"value":{"ack-mode":"on-confirm","trust-user-id":false,"uri":"amqps://federation:[email protected]?certfile=/etc/rabbitmq-tls/tls.crt&keyfile=/etc/rabbitmq-tls/tls.key&verify=verify_peer"},"vhost":"upstream","component":"federation-upstream","name":"spike"}],"global_parameters":[{"name":"cluster_name","value":"upstream"},{"name":"internal_cluster_id","value":"rabbitmq-cluster-id-xKURl_Z4LyQqc1SlYvT5lw"}],"policies":[{"vhost":"downstream","name":"federation-policy","pattern":"example","apply-to":"exchanges","definition":{"federation-upstream":"example"},"priority":1}],"queues":[{"name":"qq1","vhost":"downstream","durable":true,"auto_delete":false,"arguments":{"x-queue-type":"quorum"}},{"name":"cq1","vhost":"downstream","durable":true,"auto_delete":false,"arguments":{"x-queue-type":"classic"}},{"name":"federation: example -> upstream:downstream:example","vhost":"upstream","durable":true,"auto_delete":false,"arguments":{"x-internal-purpose":"federation"}},{"name":"qq2","vhost":"upstream","durable":true,"auto_delete":false,"arguments":{"x-queue-type":"quorum"}}],"exchanges":[{"name":"example","vhost":"downstream","type":"fanout","durable":true,"auto_delete":false,"internal":false,"arguments":{}},{"name":"federation: example -> upstream:downstream:example B","vhost":"upstream","type":"x-federation-upstream","durable":true,"auto_delete":true,"internal":true,"arguments":{"x-downstream-name":"upstream","x-downstream-vhost":"downstream","x-internal-purpose":"federation","x-max-hops":1}},{"name":"example","vhost":"upstream","type":"fanout","durable":true,"auto_delete":false,"internal":false,"arguments":{}}],"bindings":[{"source":"example","vhost":"downstream","destination":"cq1","destination_type":"queue","routing_key":"","arguments":{}},{"source":"example","vhost":"downstream","destination":"qq1","destination_type":"queue","routing_key":"","arguments":{}},{"source":"example","vhost":"upstream","destination":"federation: example -> upstream:downstream:example B","destination_type":"exchange","routing_key":"","arguments":{"x-bound-from":[{"cluster-name":"upstream","exchange":"downstream:example B","hops":1,"vhost":"downstream"}]}},{"source":"example","vhost":"upstream","destination":"qq2","destination_type":"queue","routing_key":"","arguments":{}},{"source":"federation: example -> upstream:downstream:example B","vhost":"upstream","destination":"federation: example -> upstream:downstream:example","destination_type":"queue","routing_key":"","arguments":{}}]}

docs/examples/federation-over-tls/downstream.yaml → docs/examples/federation-over-tls/rabbitmq-without-import.yaml

@@ -1,10 +1,12 @@
 apiVersion: rabbitmq.com/v1beta1
 kind: RabbitmqCluster
 metadata:
-  name: downstream
+  name: federation
 spec:
   replicas: 1
   rabbitmq:
     additionalPlugins:
-      - rabbitmq_federation
-      - rabbitmq_federation_management
+    - rabbitmq_federation
+    - rabbitmq_federation_management
+  tls:
+    secretName: tls-secret

docs/examples/federation-over-tls/rabbitmq.yaml

@@ -0,0 +1,28 @@
+apiVersion: rabbitmq.com/v1beta1
+kind: RabbitmqCluster
+metadata:
+  name: federation
+spec:
+  replicas: 1
+  rabbitmq:
+    additionalConfig: |
+      load_definitions = /federation/definitions.json # Path to the mounted definitions file
+    additionalPlugins:
+    - rabbitmq_federation
+    - rabbitmq_federation_management
+  tls:
+    secretName: tls-secret
+  override:
+    statefulSet:
+      spec:
+        template:
+          spec:
+            containers:
+            - name: rabbitmq
+              volumeMounts:
+              - mountPath: /federation/ # filename left out intentionally
+                name: definitions
+            volumes:
+            - name: definitions
+              configMap:
+                name: definitions # Name of the ConfigMap which contains definitions you wish to import

docs/examples/federation-over-tls/setup.sh

@@ -1,36 +1,5 @@
 #!/bin/bash
 
-kubectl apply -f upstream.yaml
-kubectl apply -f downstream.yaml
+kubectl create configmap definitions --from-file=definitions.json
+kubectl apply -f certificate.yaml
 
-sleep 2
-
-kubectl wait --for=condition=Ready pod/upstream-server-0
-kubectl wait --for=condition=Ready pod/downstream-server-0
-
-UPSTREAM_USERNAME=$(kubectl get secret upstream-default-user -o jsonpath="{.data.username}" | base64 --decode)
-UPSTREAM_PASSWORD=$(kubectl get secret upstream-default-user -o jsonpath="{.data.password}" | base64 --decode)
-DOWNSTREAM_USERNAME=$(kubectl get secret downstream-default-user -o jsonpath="{.data.username}" | base64 --decode)
-DOWNSTREAM_PASSWORD=$(kubectl get secret downstream-default-user -o jsonpath="{.data.password}" | base64 --decode)
-
-kubectl exec downstream-server-0 -- rabbitmqctl set_parameter federation-upstream my-upstream "{\"uri\":\"amqps://${UPSTREAM_USERNAME}:${UPSTREAM_PASSWORD}@upstream\",\"expires\":3600001}"
-
-kubectl exec downstream-server-0 -- rabbitmqctl set_policy --apply-to exchanges federate-me "^amq\." '{"federation-upstream-set":"all"}'
-
-echo "**********************************************************"
-echo "* PLEASE RUN 'sudo kubefwd svc' TO START PORT FORWARDING *"
-echo "*              and press ENTER when ready                *"
-echo "**********************************************************"
-read
-
-UPSTREAM_RABBITMQADMIN="rabbitmqadmin -U http://upstream/ -u ${UPSTREAM_USERNAME} -p ${UPSTREAM_PASSWORD} -V /"
-DOWNSTREAM_RABBITMQADMIN="rabbitmqadmin -U http://downstream/ -u ${DOWNSTREAM_USERNAME} -p ${DOWNSTREAM_PASSWORD} -V /"
-
-$UPSTREAM_RABBITMQADMIN declare queue name=test.queue queue_type=quorum
-$UPSTREAM_RABBITMQADMIN declare binding source=amq.fanout destination=test.queue
-
-$DOWNSTREAM_RABBITMQADMIN declare queue name=test.queue queue_type=quorum
-$DOWNSTREAM_RABBITMQADMIN declare binding source=amq.fanout destination=test.queue
-
-$UPSTREAM_RABBITMQADMIN publish exchange=amq.fanout routing_key=test payload="hello, world"
-$DOWNSTREAM_RABBITMQADMIN get queue=test.queue ackmode=ack_requeue_false

docs/examples/federation-over-tls/test.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+set -ex
+kubectl exec -it federation-server-0 -- rabbitmqadmin --username admin --password admin \
+  --vhost=upstream publish exchange=example routing_key=123 payload="1234"
+
+kubectl exec -it federation-server-0 -- rabbitmqadmin --username admin --password admin \
+  --vhost=downstream --format=pretty_json get queue=qq1 ackmode='ack_requeue_false' \
+  | jq -e '.[].payload'
+
+kubectl exec -it federation-server-0 -- rabbitmqadmin --username admin --password admin \
+  --vhost=downstream --format=pretty_json get queue=cq1 ackmode='ack_requeue_false' \
+  | jq -e '.[].payload'
+

docs/examples/import-definitions/definitions.json

@@ -0,0 +1 @@
+{"rabbit_version":"3.8.11","rabbitmq_version":"3.8.11","product_name":"RabbitMQ","product_version":"3.8.11","users":[{"name":"hello-world","password_hash":"JQ6+ZVMAIIpmGS/pXb9Q6elneY94TrchYGYJAKE9wtRiIpRt","hashing_algorithm":"rabbit_password_hashing_sha256","tags":"administrator","limits":{}},{"name":"guest","password_hash":"X5L0vwDQq2g8bu2Rr3oGc+uJiU+tRSFqSOj14w6zYqRK/lDU","hashing_algorithm":"rabbit_password_hashing_sha256","tags":"administrator","limits":{}}],"vhosts":[{"name":"hello-world"},{"name":"/"}],"permissions":[{"user":"guest","vhost":"hello-world","configure":".*","write":".*","read":".*"},{"user":"guest","vhost":"/","configure":".*","write":".*","read":".*"},{"user":"hello-world","vhost":"hello-world","configure":".*","write":".*","read":".*"}],"topic_permissions":[],"parameters":[],"global_parameters":[{"name":"cluster_name","value":"rabbit@73cf1fdf05d2"},{"name":"internal_cluster_id","value":"rabbitmq-cluster-id-j-jeqGlk6rJYvqR_Tb06yw"}],"policies":[],"queues":[{"name":"qq1","vhost":"hello-world","durable":true,"auto_delete":false,"arguments":{"x-queue-type":"quorum"}},{"name":"cq1","vhost":"hello-world","durable":true,"auto_delete":false,"arguments":{"x-queue-type":"classic"}}],"exchanges":[{"name":"example","vhost":"hello-world","type":"fanout","durable":true,"auto_delete":false,"internal":false,"arguments":{}}],"bindings":[{"source":"example","vhost":"hello-world","destination":"qq1","destination_type":"queue","routing_key":"","arguments":{}},{"source":"example","vhost":"hello-world","destination":"cq1","destination_type":"queue","routing_key":"1234","arguments":{}}]}

docs/examples/import-definitions/rabbitmq.yaml

@@ -12,7 +12,7 @@ spec:
             containers:
             - name: rabbitmq
               volumeMounts:
-              - mountPath: /path/to/exported/definitions.json
+              - mountPath: /path/to/exported/ # filename left out intentionally
                 name: definitions
             volumes:
             - name: definitions

docs/examples/import-definitions/setup.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+kubectl create configmap definitions --from-file=definitions.json

docs/examples/import-definitions/test.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+pushd "$(mktemp -d)" || exit 1
+
+set -x
+kubectl exec import-definitions-server-0 -- rabbitmqadmin \
+  --format=raw_json --vhost=hello-world --username=hello-world \
+  --password=hello-world --host=import-definitions.default.svc \
+  list queues &> queues.json
+
+[[ "$(jq '.[0].name' queues.json)" == '"cq1"' ]] || exit 2
+[[ "$(jq '.[1].name' queues.json)" == '"qq1"' ]] || exit 2
+
+popd || exit 1
+

docs/examples/mtls-inter-node/setup.sh

@@ -15,7 +15,7 @@ $OPENSSL genrsa -out rabbitmq-ca-key.pem 2048
 $OPENSSL req -x509 -new -nodes -key rabbitmq-ca-key.pem -subj "/CN=mtls-inter-node" -days 3650 -reqexts v3_req -extensions v3_ca -out rabbitmq-ca.pem
 
 # Create a CA secret
-kubectl create secret tls rabbitmq-ca --cert=rabbitmq-ca.pem --key=rabbitmq-ca-key.pem  
+kubectl create secret tls rabbitmq-ca --cert=rabbitmq-ca.pem --key=rabbitmq-ca-key.pem
 
 # Create an Issuer (Cert Manager CA)
 kubectl apply -f rabbitmq-ca.yaml
@@ -26,5 +26,3 @@ kubectl apply -f rabbitmq-certificate.yaml
 # Create a configuration file for Erlang Distribution
 kubectl create configmap mtls-inter-node-tls-config --from-file=inter_node_tls.config
 
-# Deploy a RabbitMQ cluster
-kubectl apply -f rabbitmq.yaml

docs/examples/mtls-inter-node/test.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+set -ex
+kubectl exec -t mtls-inter-node-server-0 -- rabbitmq-diagnostics command_line_arguments > kubectl.out
+grep '{proto_dist,\["inet_tls"\]}' kubectl.out
+

docs/examples/multiple-disks/rabbitmq.yaml

@@ -5,6 +5,8 @@ metadata:
 spec:
   replicas: 1
   rabbitmq:
+    envConfig: |
+      RABBITMQ_QUORUM_DIR=/var/lib/rabbitmq/quorum-segments
     advancedConfig: |
       [
           {ra, [
@@ -23,9 +25,6 @@ spec:
                     name: quorum-segments
                   - mountPath: /var/lib/rabbitmq/quorum-wal
                     name: quorum-wal
-                env:
-                  - name: RABBITMQ_QUORUM_DIR
-                    value: /var/lib/rabbitmq/quorum-segments
         volumeClaimTemplates:
           - apiVersion: v1
             kind: PersistentVolumeClaim

docs/examples/multiple-disks/test.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+
+set -ex
+kubectl exec -t multiple-disks-server-0 -- rabbitmqctl environment > rabbitmq-environment.out
+
+grep 'data_dir,"/var/lib/rabbitmq/quorum-segments"' rabbitmq-environment.out
+grep "{wal_data_dir,'/var/lib/rabbitmq/quorum-wal'}" rabbitmq-environment.out
+

docs/examples/tls/certificate.yaml

@@ -0,0 +1,11 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: selfsigned-cert
+spec:
+  dnsNames:
+    - '*.tls-nodes.default.svc.cluster.local'
+  secretName: tls-secret
+  issuerRef:
+    kind: ClusterIssuer
+    name: selfsigned-issuer

docs/examples/tls/setup.sh

@@ -0,0 +1,3 @@
+
+kubectl apply -f certificate.yaml
+

docs/examples/tls/test.sh

@@ -0,0 +1,6 @@
+
+set -e
+kubectl exec -it tls-server-0 -- openssl s_client -connect tls-nodes.default.svc.cluster.local:5671 </dev/null
+
+kubectl exec -it tls-server-0 -- openssl s_client -connect tls-nodes.default.svc.cluster.local:15671 </dev/null
+