From b74e34f2e51487b61a99be94adcc137e94626543 Mon Sep 17 00:00:00 2001
From: Bipul Adhikari <badhikar@redhat.com>
Date: Fri, 15 Nov 2024 10:08:48 +0545
Subject: [PATCH] docs: add design for using services for addons pods

Signed-off-by: Bipul Adhikari <badhikar@redhat.com>
---
 docs/design/podService.md | 94 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 94 insertions(+)
 create mode 100644 docs/design/podService.md

diff --git a/docs/design/podService.md b/docs/design/podService.md
new file mode 100644
index 00000000..7ba48863
--- /dev/null
+++ b/docs/design/podService.md
@@ -0,0 +1,94 @@
+# Services for Addons Pods
+
+The Kubernetes CSI Addons project currently connects the CSI Addons Manager to the add-on sidecar through hardcoded IP addresses. However, this IP-based approach poses limitations, especially when implementing secure TLS communication between the manager and the sidecar. TLS certificates typically require stable DNS-based connections rather than IP-based, as IP addresses can change and are not easily resolved in cases where host networking is enabled.
+
+## Problem Statement
+
+1. **IP Dependency**: The current approach hardcodes IP addresses, which can be unreliable and lead to broken connections if IPs change.
+2. **TLS Challenges**: Introducing TLS certificates requires a stable DNS name for validation. Relying on IPs does not align with best practices in certificate management and TLS.
+3. **Host Networking Constraints**: Host networking limits the use of Pod DNS names, as Pod DNS entries may not resolve correctly or may be inaccessible.
+
+## Proposed Solution
+
+### 1. Exposing Add-ons Sidecar via Kubernetes Service
+
+To address the above challenges, this proposal suggests creating a dedicated Kubernetes Service for each add-ons sidecar Pod. This Service will:
+
+- Provide a consistent DNS name, eliminating the dependency on IP addresses.
+- Enable TLS certificates to be issued with a stable DNS name.
+- Allow secure connectivity between the manager and the sidecar by abstracting away the networking specifics, especially when host networking is enabled.
+
+The Service could be of type `ClusterIP`, as the primary objective is intra-cluster communication between the manager and sidecar.
+
+#### Benefits:
+
+- **Stability**: Service DNS names provide a consistent endpoint that does not change with Pod restarts or IP changes.
+- **Security**: By using a Service DNS name in the TLS certificate, we can achieve a secure, verifiable connection between the manager and sidecar.
+
+### 2. Service Name Propagation through CSI Annotations
+
+To inform the manager of the sidecar Service, we propose introducing a custom CSI annotation on the add-ons sidecar Pod. This annotation will store the Service name associated with the Pod, allowing the manager to:
+
+- Read the Service name dynamically.
+- Use the Service DNS name for TLS certificate validation.
+
+The workflow for this process:
+
+- The sidecar Pod will have an annotation, e.g., `csi.addons.service/name`, that stores the corresponding Service name.
+- The CSI Addons Manager will parse this annotation and resolve the Service DNS name.
+- The TLS certificate will then be generated using this DNS name, enabling secure and verifiable communication.
+
+### 3. Certificate Issuance Process
+
+For the manager-sidecar communication, we will use the following certificate generation workflow:
+
+- The manager will request a TLS certificate with the Service DNS name as the Common Name (CN).
+- The Service DNS name will be included in the certificate’s Subject Alternative Name (SAN) field, which will allow it to match the Service endpoint.
+- The manager will verify the sidecar's certificate using this DNS name, ensuring a secure TLS handshake without relying on changing IP addresses.
+
+## Implementation Details
+
+1. **Service Creation**: Modify the ceph-csi-operator configuration to create a Service for each sidecar Pod, with the appropriate pod-selectors.
+2. **Annotation Configuration**: Update the Pod configuration to include the `csi.addons.service/name` annotation.
+
+### CSI Addons related changes
+
+1. **Manager Logic Update**: Enhance the CSI Addons Manager logic to:
+   - Read the `csi.addons.service/name` annotation on startup or during reconnection attempts.
+   - Use the Service DNS name in the certificate verification process.
+
+### Example YAML Configurations
+
+#### Service Definition
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: addon-sidecar-service
+  namespace: <namespace>
+spec:
+  selector:
+    app: addon-sidecar
+  ports:
+    - protocol: TCP
+      port: 8443
+      targetPort: 8443
+  type: ClusterIP
+```
+
+#### Pod annotations
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: addon-sidecar
+  namespace: <namespace>
+  annotations:
+    csi.addons.service/name: addon-sidecar-service
+spec:
+  containers:
+    - name: addon-sidecar
+      image: <sidecar-image>
+```
\ No newline at end of file