Feature Rook/Ceph storage (#3002)

* Define additional disks with defined sizes to VMs for Azure (#2953) * Additional disks for azure DRAFT * Simplify config * add changelog note * Change starting index value in names * Change value of lun attribute to alling with disks indexing * Merge 'develop' into 'feature/rook-ceph-storage' (#2959) * K8s improvements (#2918) * Removal of Logstash component - next steps #2833 (#2941) * Add information about manual package removal that needs to be performed by cluster admin * include information about inventory file * Fix for ansible_default_ipv4 empty value (#2922) * Fix for ansible_default_ipv4 empty value * Fix after ToBar review * Add check if there is only one default routing config * Add check for different metrics value in case more than one default routing * Review fixes * Update fail_msg Co-authored-by: to-bar <[email protected]> * Extend k8s schema validation (#2929) * Extend deprecation note with Ignite removal steps (#2949) * Escalate permissions for ip command (#2952) * Changed autoscaling_group to plain EC2 VMs on AWS. (#2939) - Replaced AWS auto_scaling_groups with plain EC2 VM creation. - Added proper host sorting how it was implemented for the any and azure providers: #1076 - Sync up features with Azure Terraform implementation - Added support for use_network_security_groups flag #959 - Updated DoD for bugs to reflect changes made for #2832 Co-authored-by: Anatoli Tsikhamirau <[email protected]> Co-authored-by: Irek Głownia <[email protected]> Co-authored-by: to-bar <[email protected]> Co-authored-by: Luuk van Venrooij <[email protected]> * Add Ansible Rook role (#2960) * Add initial ansible rook role * Fix issue with no yet existing template * Add Rook schema related files * Remove already setup vars, add missing enters * Fix issue with feature mapping * Add configuration support * Move kubeconfig to rook role yaml * Update changelog * Add started dashes to yaml defaults for Rook * Add RH fix suggested by cicharka * Add rook cluster helm chart installation * Add separate chart values definitions * Limit fact checking in rook role * Add initial rook/ceph documentation * Update documentation for Rook * Update download requirements * Persistent Storage: Note about disks on AWS (#2998) * Updated changelog * Rearrange and update documentation * Update components * Fix after review * Limit hosts where apply is specified * Add comment about repository facts gathering * Make rook namespace configurable * Add linter comments to disable false positive * Add requested comments * Change shell to command module * Change mode to preserve for temp configuration data * Update documentation * Update ansible/playbooks/roles/repository/files/download-requirements/requirements/x86_64/images.yml Co-authored-by: przemyslavic <[email protected]> * Update ansible/playbooks/roles/repository/files/download-requirements/requirements/x86_64/images.yml Co-authored-by: przemyslavic <[email protected]> * Update ansible/playbooks/roles/repository/files/download-requirements/requirements/x86_64/images.yml Co-authored-by: przemyslavic <[email protected]> * Update ansible/playbooks/roles/repository/files/download-requirements/requirements/x86_64/images.yml Co-authored-by: przemyslavic <[email protected]> * Update ansible/playbooks/roles/repository/files/download-requirements/requirements/x86_64/images.yml Co-authored-by: przemyslavic <[email protected]> * Update ansible/playbooks/roles/repository/files/download-requirements/requirements/x86_64/images.yml Co-authored-by: przemyslavic <[email protected]> * Update ansible/playbooks/roles/repository/files/download-requirements/requirements/x86_64/images.yml Co-authored-by: przemyslavic <[email protected]> * Update ansible/playbooks/roles/repository/files/download-requirements/requirements/x86_64/images.yml Co-authored-by: przemyslavic <[email protected]> * use newer version of rook (#3053) * Feature/rook ceph storage additions (#3064) * rook/ceph: add default values in epiphany configuration * rook/ceph (#3068) * add default values in epiphany configuration * bump up verison of rook * Revert schema defaults (#3074) * Revert "rook/ceph (#3068)" This reverts commit 989be85. * Revert "Feature/rook ceph storage additions (#3064)" This reverts commit 17c49af. * minor fixes for schema and docs * add rook feature_mapping for single and custom * rook: modify tests and schema for image registry * rook: offline mode * enahnce playbook to allow chart files use images from local image registry * remove unused condition for chart and operator specification which is always defined now * modify path for rook/ceph image in order to be compliant with values.yml Signed-off-by: cicharka <[email protected]> * rook: add missing csiaddons for offline mode Signed-off-by: cicharka <[email protected]> Co-authored-by: Anatoli Tsikhamirau <[email protected]> Co-authored-by: Irek Głownia <[email protected]> Co-authored-by: to-bar <[email protected]> Co-authored-by: Luuk van Venrooij <[email protected]> Co-authored-by: cicharka <[email protected]> Co-authored-by: przemyslavic <[email protected]> Co-authored-by: cicharka <[email protected]>
hitachienergy · Apr 20, 2022 · bf36a7f · bf36a7f
1 parent 3cb0f0c
commit bf36a7f
Show file tree

Hide file tree

Showing 18 changed files with 400 additions and 8 deletions.
diff --git a/ansible/playbooks/roles/repository/files/download-requirements/requirements/x86_64/files.yml b/ansible/playbooks/roles/repository/files/download-requirements/requirements/x86_64/files.yml
@@ -38,3 +38,9 @@ files:
 
   'https://helm.elastic.co/helm/filebeat/filebeat-7.9.2.tgz':
     sha256: 5140b4c4473ca33a0af4c3f70545dcc89735c0a179d974ebc150f1f28ac229ab
+
+  'https://charts.rook.io/release/rook-ceph-v1.8.8.tgz':
+    sha256: f67e474dedffd4004f3a0b7b40112694a7f1c2b1a0048b03b3083d0a01e86b14
+
+  'https://charts.rook.io/release/rook-ceph-cluster-v1.8.8.tgz':
+    sha256: df4e1f2125af41fb84c72e4d12aa0cb859dddd4f37b3d5979981bd092040bd16
diff --git a/...ble/playbooks/roles/repository/files/download-requirements/requirements/x86_64/images.yml b/...ble/playbooks/roles/repository/files/download-requirements/requirements/x86_64/images.yml
@@ -157,3 +157,34 @@ images:
 
   'calico/pod2daemon-flexvol:v3.20.3':
     sha1: 97c1b7ac90aa5a0f5c52e7f137549e598ff80f3e
+
+  # --- Rook ---
+  'k8s.gcr.io/sig-storage/csi-attacher:v3.4.0':
+    sha1: f076bd75359c6449b965c48eb8bad96c6d40790d
+
+  'k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0':
+    sha1: 129eb73c8e118e5049fee3d273b2d477c547e080
+
+  'k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0':
+    sha1: 2b45e5a3432cb89f3aec59584c1fa92c069e7a38
+
+  'k8s.gcr.io/sig-storage/csi-resizer:v1.4.0':
+    sha1: ce5c57454254c195762c1f58e1d902d7e81ea669
+
+  'k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1':
+    sha1: be1cf43617eea007629c0eb99149a99b6498f889
+
+  'quay.io/ceph/ceph:v16.2.7':
+    sha1: 13275be4d347e9b305608f52a544e068a73e949f
+
+  'quay.io/cephcsi/cephcsi:v3.5.1':
+    sha1: 51dee9ea8ad76fb95ebd16f951e8ffaaaba95eb6
+
+  'quay.io/csiaddons/k8s-sidecar:v0.2.1':
+    sha1: f0fd757436ac5075910c460c1991ff67c4774d09
+
+  'quay.io/csiaddons/volumereplication-operator:v0.3.0':
+    sha1: d3cd17f14fcbf09fc6c8c2c5c0419f098f87a70f
+
+  'rook/ceph:v1.8.8':
+    sha1: f34039b17b18f5a855b096d48ff787b4013615e4
diff --git a/ansible/playbooks/roles/rook/defaults/main.yml b/ansible/playbooks/roles/rook/defaults/main.yml
@@ -0,0 +1,5 @@
+---
+rook_helm_chart_file_name: rook-ceph-v1.8.8.tgz
+rook_helm_cluster_chart_file_name: rook-ceph-cluster-v1.8.8.tgz
+rook_helm_chart_name: rook-ceph
+rook_helm_cluster_chart_name: rook-ceph-cluster
diff --git a/ansible/playbooks/roles/rook/tasks/main.yml b/ansible/playbooks/roles/rook/tasks/main.yml
@@ -0,0 +1,82 @@
+---
+- name: Prepare configuration and upgrade/install Rook Helm chart
+  when: specification.enabled
+  become: true
+  run_once: true
+  block:
+    - name: RedHat fix | Create helm's binary symlink
+      file:
+        src: "/usr/local/bin/helm"
+        dest: "/usr/bin/helm"
+        state: link
+      when: ansible_os_family == 'RedHat'
+
+    - name: Download Rook's Chart Files
+      include_role:
+        name: download
+        tasks_from: download_file
+      vars:
+        file_name: "{{ item }}"
+      loop:
+        - "{{ rook_helm_chart_file_name }}"
+        - "{{ rook_helm_cluster_chart_file_name }}"
+
+    - name: Decide if internal docker registry will be used
+      set_fact:
+        use_epiphany_image_registry: >-
+          {{ _use_local_image_registry }}
+      vars:
+        _use_local_image_registry: >-
+          {{ (specification.use_local_image_registry is undefined) or (specification.use_local_image_registry | bool) }}
+
+    - name: Adjust image paths in charts to use internal docker registry
+      when: use_epiphany_image_registry
+      block:
+        - name: Adjust images paths in operator_chart_values
+          set_fact:
+            operator_images: "{{ lookup('template', 'operator-images.yml.j2') | from_yaml }}"
+          vars:
+            data: "{{ specification.operator_chart_values }}"
+
+        - name: Adjust images paths in cluster_chart_values
+          set_fact:
+            cluster_images: "{{ lookup('template', 'cluster-images.yml.j2') | from_yaml }}"
+          vars:
+            data: "{{ specification.cluster_chart_values }}"
+
+    - name: Create custom configuration for operator Helm chart file (operator-custom-chart-values.yml)
+      copy:
+        content: |-
+          {% if use_epiphany_image_registry %}
+          {{ specification.operator_chart_values | combine(operator_images, recursive=True) }}
+          {% else %}
+          {{ specification.operator_chart_values }}
+          {% endif %}
+        dest: "{{ download_directory }}/operator-custom-chart-values.yml"
+        mode: preserve
+
+    - name: Create custom configuration for cluster Helm chart file (cluster-custom-chart-values.yml)
+      copy:
+        content: |-
+          {% if use_epiphany_image_registry %}
+          {{ specification.cluster_chart_values | combine(cluster_images, recursive=True) }}
+          {% else %}
+          {{ specification.cluster_chart_values }}
+          {% endif %}
+        dest: "{{ download_directory }}/cluster-custom-chart-values.yml"
+        mode: preserve
+
+    - name: Install Rook operator using Helm chart with values from operator-custom-chart-values.yml
+      command: |
+        helm -n {{ specification.rook_namespace }} upgrade --install \
+          -f {{ download_directory }}/operator-custom-chart-values.yml \
+          {{ rook_helm_chart_name }} \
+          {{ download_directory }}/{{ rook_helm_chart_file_name }} --create-namespace
+
+    - name: Create Rook cluster with values from cluster-custom-chart-values.yml
+      command: |
+        helm -n {{ specification.rook_namespace }} upgrade --install \
+          --set operatorNamespace={{ specification.rook_namespace }} \
+          -f {{ download_directory }}/cluster-custom-chart-values.yml \
+          {{ rook_helm_cluster_chart_name }} \
+          {{ download_directory }}/{{ rook_helm_cluster_chart_file_name }} --create-namespace
diff --git a/ansible/playbooks/roles/rook/templates/cluster-images.yml.j2 b/ansible/playbooks/roles/rook/templates/cluster-images.yml.j2
@@ -0,0 +1,5 @@
+toolbox:
+  image: {{ image_registry_address }}/{{ data.toolbox.image }}
+cephClusterSpec:
+  cephVersion:
+    image: {{ image_registry_address }}/{{ data.cephClusterSpec.cephVersion.image }}
diff --git a/ansible/playbooks/roles/rook/templates/operator-images.yml.j2 b/ansible/playbooks/roles/rook/templates/operator-images.yml.j2
@@ -0,0 +1,8 @@
+image:
+  repository: {{ image_registry_address }}/{{ data.image.repository }}
+  tag: {{ data.image.tag }}
+csi:
+{% for name, value in data.csi.items() %}
+  {{ name }}:
+    image: {{ image_registry_address }}/{{ value.image }}
+{% endfor %}
diff --git a/ansible/playbooks/rook.yml b/ansible/playbooks/rook.yml
@@ -0,0 +1,13 @@
+---
+# Added to solve an issue with finding the repository server by download rook helm chart task
+- hosts: repository
+  gather_facts: true
+  tasks: []
+
+- hosts: kubernetes_master[0]
+  become: true
+  become_method: sudo
+  roles:
+    - rook
+  environment:
+    KUBECONFIG: "{{ kubeconfig.remote }}"
diff --git a/docs/changelogs/CHANGELOG-2.0.md b/docs/changelogs/CHANGELOG-2.0.md
@@ -20,6 +20,12 @@
 - [#2448](https://github.com/epiphany-platform/epiphany/issues/2448) - Passwordless SSH communication for postgres user between DB nodes
 - [#2821](https://github.com/epiphany-platform/epiphany/issues/2821) - Node Exporter preflight checks
 - [#2996](https://github.com/epiphany-platform/epiphany/issues/2996) - Introduce the new configuration field to change a component name
+- [#2888](https://github.com/epiphany-platform/epiphany/issues/2888) - Define additional disks with defined sizes to VMs for Azure
+- [#2812](https://github.com/epiphany-platform/epiphany/issues/2812) - Extend K8s config validation
+- [#2890](https://github.com/epiphany-platform/epiphany/issues/2890) - Rook/Ceph Kubernetes Native Storage for Azure Kubernetes non-AKS cluster
+- [#1452](https://github.com/epiphany-platform/epiphany/issues/1452) - Create Kubernetes Native Storage Epiphany configuration for Azure provider
+- [#2891](https://github.com/epiphany-platform/epiphany/issues/2891) - Rook/Ceph Kubernetes Native Storage for AWS non-EKS
+- [#2887](https://github.com/epiphany-platform/epiphany/issues/2887) - Create Kubernetes Native Storage Epiphany configuration for AWS provider
 
 ### Fixed
 

diff --git a/docs/home/COMPONENTS.md b/docs/home/COMPONENTS.md
@@ -39,6 +39,7 @@ Note that versions are default versions and can be changed in certain cases thro
 | Pgpool                     | 4.2.4    | https://www.pgpool.net/                               | [License](https://www.pgpool.net/mediawiki/index.php/pgpool-II_License) |
 | Alertmanager               | 0.23.0   | https://github.com/prometheus/alertmanager            | [Apache License 2.0](https://github.com/prometheus/alertmanager/blob/master/LICENSE) |
 | Apache2                    | 2.4.29   | https://httpd.apache.org/                             | [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) |
+| Rook                    | 1.8.8   | https://rook.io/                             | [Apache License 2.0](https://github.com/rook/rook/blob/master/LICENSE) |
 
 ## Epicli binary dependencies
 

diff --git a/docs/home/howto/kubernetes/PERSISTENT_STORAGE.md b/docs/home/howto/kubernetes/PERSISTENT_STORAGE.md
@@ -1,8 +1,140 @@
-## Kubernetes persistent storage
+# Kubernetes Persistent Storage
 
-Epiphany supports [Azure Files](https://docs.microsoft.com/en-us/azure/storage/files/storage-files-introduction)
-and [Amazon EFS](https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html) storage types to use as Kubernetes
-persistent volumes.
+In Epiphany there are two supported ways of setting up Kubernetes Persistent Storage:
+- Rook/Ceph Cluster Storage with disks resources created by Epiphany
+- [Azure Files](https://docs.microsoft.com/en-us/azure/storage/files/storage-files-introduction)
+or [Amazon EFS](https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html) storage types to use as Kubernetes persistent volumes
+
+## Kubernetes Rook/Ceph Cluster Storage
+
+Rook provides distributed storage systems for Kubernetes installed with Epiphany.
+It provides capabilities:
+- self-managing
+- self-scaling
+- self-healing
+- upgrading
+- migration
+- disaster recovery
+- monitoring
+
+Epiphany supports Rook with Ceph storage, other options provided by Rook - Cassandra, NFS are not supported.
+
+### Rook/Ceph General Configuration
+
+To add Rook/Ceph support in Epiphany you need to add to your cluster configuration two elements:
+- Storage (for cloud deployments - can be automatically created by Epiphany)
+- Rook/Ceph
+
+Adding the storage is described below in separate sections for Azure, AWS and on premise environments. Rook/Ceph configuration in Epiphany is described after add disk paragraphs.
+
+#### Create disks for Rook/Ceph Cluster Storage - Azure
+
+To create Rook/Ceph Cluster Storage on Azure first you need to add empty disk resource to Kubernetes cluster in key `specification.additional_disks`, under `kind: infrastructure/virtual-machine` for configuration of kubernetes node machine:
+
+```yaml
+---
+kind: infrastructure/virtual-machine
+name: kubernetes-node-machine
+provider: azure
+based_on: kubernetes-node-machine
+specification:
+  storage_image_reference:
+    ..
+  storage_os_disk:
+    disk_size_gb: 64
+  additional_disks:
+    - storage_account_type: Premium_LRS
+      disk_size_gb: 128
+```
+
+#### Create disks for Rook/Ceph Cluster Storage - AWS
+
+To define additional empty disk resources for Rook/Ceph Cluster Storage on AWS, use `specification.disks.additional_disks` under `kind: infrastructure/virtual-machine` for configuration of kubernetes node machine:
+```yaml
+---
+kind: infrastructure/virtual-machine
+title: Virtual Machine Infra
+provider: aws
+name: kubernetes-node-machine
+specification:
+  disks:
+    additional_disks:
+      - device_name: "/dev/sdb"
+        volume_type: gp2
+        volume_size: 64
+        delete_on_termination: false
+        encrypted: true
+```
+Currently Epiphany support the following parameters for `additional_disks` specification:
+- device_name
+- volume_type
+- volume_size
+- encrypted
+- delete_on_termination
+- tags
+
+More information about AWS block devices and its parameters: [ebs_block_device](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance#ebs_block_device)
+
+
+#### Create disks for Rook/Ceph Cluster Storage - On Prem
+
+To add disks to Rook/Ceph Cluster Storage you need to attach first raw devices to Kubernetes nodes machines and all raw devices will be used as Rook/Ceph devices.
+
+#### Rook/Ceph Configuration
+
+To enable Rook support add to your cluster configuration the section like below:
+
+```yaml
+---
+kind: configuration/rook
+title: "Kubernetes Rook Config"
+name: default
+specification:
+    enabled: true
+```
+
+If you want to install rook and rook cluster in the namespace different than `rook-ceph`, you need to add key `rook_namespace` with desired namespace name as value like in the sample below.
+
+```yaml
+---
+kind: configuration/rook
+title: "Kubernetes Rook Config"
+name: default
+specification:
+    enabled: true
+    rook_namespace: your-rook-namespace
+```
+
+The key `specification.enabled` must be set to true to install Rook/Ceph component. Epiphany configuration file provides set of
+parameters that are used for Rook/Ceph installation with default values. To override default values provided by Rook you need
+to adjust `configuration/rook` keys:
+- `specification.operator_chart_values` - to override Rook Operator Helm Chart default values
+- `specification.cluster_chart_values` - to override Rook Cluster Helm Chart default values
+
+```yaml
+---
+kind: configuration/rook
+title: "Kubernetes Rook Config"
+name: default
+specification:
+    enabled: true
+    operator_chart_values: |
+      ...
+    cluster_chart_values: |
+      ...
+```
+Values nested below the `operator_chart_values` and `cluster_chart_values` keys are respectively Helm Chart values for Rook Operator and Rook Ceph Cluster.
+It is important to ensure that configuration of operator and chart values matches configuration of your cluster.
+
+More information about Helm Chart values may be found:
+- [Helm Operator](https://github.com/rook/rook/blob/master/Documentation/helm-operator.md)
+- [Helm Ceph Cluster](https://github.com/rook/rook/blob/master/Documentation/helm-ceph-cluster.md)
+
+Sample configuration files that can be used in Epiphany `configuration/rook`:
+- [Helm Operator](https://raw.githubusercontent.com/rook/rook/v1.8.8/deploy/charts/rook-ceph/values.yaml)
+- [Helm Ceph Cluster](https://raw.githubusercontent.com/rook/rook/v1.8.8/deploy/charts/rook-ceph-cluster/values.yaml)
+
+More informations about Rook with Ceph storage may be found in the official Rook [documentation](https://rook.io/docs/rook/v1.8/).
 
 ### Azure
 

diff --git a/schema/azure/defaults/infrastructure/virtual-machine.yml b/schema/azure/defaults/infrastructure/virtual-machine.yml
@@ -27,6 +27,9 @@ specification:
     create_option: FromImage
     disk_size_gb: 32
     managed_disk_type: Premium_LRS
+  additional_disks: []
+    # - storage_account_type: Premium_LRS
+    #   disk_size_gb: 32
   network_interface:
     enable_accelerated_networking: false
     private_ip:

diff --git a/schema/azure/validation/infrastructure/virtual-machine.yml b/schema/azure/validation/infrastructure/virtual-machine.yml
@@ -55,6 +55,15 @@ properties:
         type: integer
       managed_disk_type:
         type: string
+  additional_disks:
+    type: array
+    items:
+      type: object
+      properties:
+        storage_account_type:
+          type: string
+        disk_size_gb:
+          type: integer
   network_interface:
     type: object
     properties:

diff --git a/schema/common/defaults/configuration/feature-mapping.yml b/schema/common/defaults/configuration/feature-mapping.yml
@@ -49,6 +49,8 @@ specification:
       enabled: true
     - name: applications
       enabled: true
+    - name: rook
+      enabled: true
 
   roles_mapping:
     kafka:
@@ -100,6 +102,7 @@ specification:
       - node-exporter
       - logging
       - firewall
+      - rook
     single_machine:
       - repository
       - image-registry
@@ -109,10 +112,12 @@ specification:
       - rabbitmq
       - postgresql
       - firewall
+      - rook
     kubernetes_master:
       - kubernetes-master
       - helm
       - applications
+      - rook
       - node-exporter
       - filebeat
       - firewall