Release 1.0.4-GA (#155)

* #0000 feat azure terraform changes (#150)

* Local Installation Scripts (#151)

* Sanketika-Obsrv/issue-tracker#26: Helm Changes for Local Setup

* Sanketika-Obsrv/issue-tracker#26: CRD's for Local Setup

* delete cloudwatch loggroups when using terraform destroy

* Sanketika-Obsrv/issue-tracker#65: fix:Bug fix terraform destroy deletes cloudwatch log group

* Sanketika-Obsrv/issue-tracker#65: fix:Bug fix terraform destroy deletes cloudwatch log group

* Sanketika-Obsrv/issue-tracker#65: fix:Bug fix terraform destroy deletes cloudwatch log group

* Add connector info into system events ingestion spec. (#153)

* #8: Update Documentation for AWS and Azure (#154)

* Sanketika-Obsrv/issue-tracker#8: doc: update shell script documentation for infra setup

* Sanketika-Obsrv/issue-tracker#8: doc: update azure documentation

* Sanketika-Obsrv/issue-tracker#8: doc: update AWS documentation for velero backup

* Sanketika-Obsrv/issue-tracker#8: fix: typo in installation doc

---------

Co-authored-by: Manoj Krishna <[email protected]>
Co-authored-by: Ravi Mula <[email protected]>
Co-authored-by: divyagovindaiah <[email protected]>
Co-authored-by: Sowmya N Dixit <[email protected]>

INSTALLATION.md

@@ -39,6 +39,7 @@ Prerequisites:
     export AWS_TERRAFORM_BACKEND_BUCKET_NAME=mybucket
     export AWS_TERRAFORM_BACKEND_BUCKET_REGION=myregion
     ```
+- You will need `velero cli` to create the cluster backups. Learn how to install velero cli at ([Velero cli](https://velero.io/docs/v1.3.0/velero-install/))
 
 #### Steps:
 * Execute the below steps in the same terminal session:
@@ -51,9 +52,24 @@ Prerequisites:
     terragrunt apply -target=module.get_kubeconfig -auto-approve
     terragrunt apply
     ```
-The installer will ask for user inputs twice:
- - Before creating the EKS cluster
- - Before creating rest of the components
+    The installer will ask for user inputs twice:
+    - Before creating the EKS cluster
+    - Before creating rest of the components
+
+* Create a velero backup:
+    - After the cluster is created velero backup needs to be triggered manually
+    - We need to create a backup and schedule manually
+    - Run the below commands to create a backup and schedule
+        ```bash
+        velero backup create <backup_name>
+        velero backup schedule <backup_schedule_name>
+        ```
+    - Below example Creates a backup and schedule it for every 24h and retain the backup for 50h
+        ```bash
+        velero backup create obsrv-dev-full-cluster-backup
+        velero backup schedule obsrv-dev-full-cluster-daily-backup --schedule="@every 24h" --ttl 50h0m0s
+        ```
+
 
 #### Tip:
 Add `-auto-approve` to the above `terragrunt` command to install without providing user inputs as shown below
@@ -64,8 +80,8 @@ terragrunt apply -target=module.eks -auto-approve && terragrunt apply -target=mo
 **Azure**
 ### Prerequisites:
 * Log into your cloud environment in your terminal. Please see [Sign in with Azure CLI](https://learn.microsoft.com/en-us/cli/azure/authenticate-azure-cli) for reference.
-    ```
-    az login
+    ``` bash
+    az login --allow-no-subscriptions
     ```        
 * Create a storage account and export the below variables in your terminal. Please see [Create a storage container](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-create?toc=/azure/storage/blobs/toc.json) for reference. Export the below variables in your terminal session
     ```
@@ -75,12 +91,35 @@ terragrunt apply -target=module.eks -auto-approve && terragrunt apply -target=mo
     ```
 ### Steps:
 * Execute the below commands in the same terminal session: 
-    ```
+    ```bash
     cd terraform/azure
     terragrunt init
-    terragrunt plan
-    terragrunt apply
+    terragrunt apply -target module.aks -auto-approve
+    ```
+* Pass the following variables when prompted:
+    ```bash
+    env: dev
+    building_block: obsrv
+    location: East US 2
+    ```
+- Note:  All the above variable values are given for example
+* Export the below variables:
+    ``` bash
+    export KUBE_CONFIG_PATH=<path_to_kubeconfig>( default to current directory)
+    export KUBECONFIG=<path_to_kubeconfig>( default to current directory)
     ```
+* Execute the below commands in the same terminal session:
+    ``` bash
+    terragrunt apply -auto-approve
+    ```
+* Pass the following variables when prompted:
+    ```bash
+    env: dev
+    building_block: obsrv
+    location: East US 2
+    ```
+- Note:  All the above variable values are given for example
+
 **GCP**
 ### Prerequisites:
 * Setup the gcoud CLI. Please see [Installing Google Cloud SDK](https://cloud.google.com/sdk/docs/install) for reference.

automation-scripts/infra-setup/setup.md → automation-scripts/infra-setup/obsrv.md

@@ -3,7 +3,7 @@
 
 ## Configuration
 
-Define the necessary configurations in the `setup.conf` file:
+Define the necessary configurations in the `obsrv.conf` file:
 
 ### setup.conf
 
@@ -34,20 +34,12 @@ Ensure the installation of the following tools:
 
 ## Setup Process
 
-Before executing the `setup` shell script, ensure that the `curl` and `unzip` utilities are present on your system. If they are not installed, you can use the following commands to install them. Execute the provided `setup.sh`` script as a root user to avoid potential permission issues:
-
-**Prerequisites**
-```bash
-sudo apt-get update
-sudo apt-get install -y curl
-sudo apt-get install -y unzip
-```
-
-Once the `curl` and `unzip` utility is successfully installed, proceed with the setup by running the following command:
-
-```bash
-sh setup.sh ./setup.conf
-```
-
-This command utilizes the configurations specified in the `setup.conf` file to initiate the setup process.
+1. Run Installation Script: Execute the following command to start the installation process:
+    - Before installing please provide executable permission to installation script
+    `chmod +x ./obsrv.sh`
+    ```bash
+    ./obsrv.sh install --config ./obsrv.conf --install_dependencies false
+    ```
+    - Note: Setting install_dependencies=true will automatically download and install all required dependencies. If preferred, you can manually download the dependencies instead.
+2. Monitor Installation Progress: The script will begin installing Obsrv within the AWS cluster. Monitor the progress and follow any on-screen prompts or instructions.
 

terraform/azure/main.tf

@@ -51,4 +51,14 @@ provider "helm" {
     client_key             = module.aks.client_key
     cluster_ca_certificate = module.aks.cluster_ca_certificate
   }
+}
+
+module "unified_helm" {
+  source                      = "../modules/helm/unified_helm"
+  building_block              = var.building_block
+  env                         = var.env
+  depends_on                  = [ module.storage,module.aks ]
+  azure_storage_account_key   = module.storage.azurerm_storage_account_key
+  azure_storage_account_name  = module.storage.azurerm_storage_account_name
+  azure_storage_container     = module.storage.azurerm_storage_container
 }

terraform/azure/variables.tf

@@ -23,40 +23,4 @@ variable "druid_deepstorage_type" {
     type        = string
     description = "Druid deep strorage type."
     default     = "azure"
-}
-
-variable "flink_checkpoint_store_type" {
-    type        = string
-    description = "Flink checkpoint store type."
-    default     = "azure"
-}
-
-variable "dataset_api_container_registry" {
-  type        = string
-  description = "Container registry. For example docker.io/obsrv"
-  default     = "sanketikahub"
-}
-
-variable "dataset_api_image_tag" {
-  type        = string
-  description = "Dataset api image tag."
-  default     = "1.0.0"
-}
-
-variable "flink_container_registry" {
-  type        = string
-  description = "Container registry. For example docker.io/obsrv"
-  default     = "manjudr"
-}
-
-variable "flink_image_tag" {
-   type        = string
-   description = "Flink kubernetes service name."
-   default     = "1.1.0"
-}
-
-variable "command_service_image_tag" {
-  type        = string
-  description = "CommandService image tag."
-  default     = "1.0.0"
 }