Develop to 1.0.2-GA (Sanketika-Obsrv#140) (Sanketika-Obsrv#141)

* Fix: removed docker secrets

* updated the grafana_secrete

* Feat: updated image tags according to release 1.1.0

* Issue Sanketika-Obsrv#171 feat: update job to deployment for jobmanagers

* Sanketika-Obsrv#90 fix: Add default value for published_date in tables (Sanketika-Obsrv#94)

* fix: remove restart policy from flink deployment manifest (Sanketika-Obsrv#96)

* Sanketika-Obsrv#196 Fix:Remove default region. (Sanketika-Obsrv#95)

* fix: Sanketika-Obsrv#196 remove default value for region

* update: Sanketika-Obsrv#196 missing variable in variables.tf file

* Sanketika-Obsrv#196 update:  example on how provide region

* Sanketika-Obsrv#196 update: comments on how to provide region

* Separation of redis into dedup & meta/denorm. (Sanketika-Obsrv#92)

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

* develop into release 1.3.0 (Sanketika-Obsrv#97)

* Sanketika-Obsrv#90 fix: Add default value for published_date in tables (Sanketika-Obsrv#94)

* fix: remove restart policy from flink deployment manifest (Sanketika-Obsrv#96)

* Sanketika-Obsrv#196 Fix:Remove default region. (Sanketika-Obsrv#95)

* fix: Sanketika-Obsrv#196 remove default value for region

* update: Sanketika-Obsrv#196 missing variable in variables.tf file

* Sanketika-Obsrv#196 update:  example on how provide region

* Sanketika-Obsrv#196 update: comments on how to provide region

* Separation of redis into dedup & meta/denorm. (Sanketika-Obsrv#92)

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

* Sanketika-Obsrv#58 feat: Separation of redis into dedup & meta/denorm.

---------





* Sanketika-Obsrv#126 feat: add new property to datasources table (Sanketika-Obsrv#99)

* Sanketika-Obsrv#126 feat: add new property to datasources table

* Sanketika-Obsrv#126 fix: allow property to be nullable

* #0000 - feat: Fix masterdata stats ingestion spec (Sanketika-Obsrv#100)

* #0000 - feat: Fix masterdata stats ingestion spec
* #0000 - feat: Add latency and processing times

* #0000 - feat: Flink helm configuration changes (Sanketika-Obsrv#103)

* #0000 - feat: Flink helm configuration changes

* #0000 - feat: Ingestion spec changes for system-events

* #0000 - feat: Remove outdated ingestion specs

* #0000 - feat: Secor changes for unified failed topic (Sanketika-Obsrv#104)

* feat: azure terraform changes to provision cluster (Sanketika-Obsrv#106)

* feat: azure terraform changes to provision cluster

* cleanup azure terraform code

* update node count and instance size for azure (Sanketika-Obsrv#107)

* Issue #305 feat: exclude system-events for the datasource validation (Sanketika-Obsrv#108)

* Issue #0000 fix: system events ingestion spec typo fix (Sanketika-Obsrv#105)

* Issue #0000 fix: system events ingestion spec typo fix

* Issue #0000 fix: system-events ingestion spec update to add dataset_type

* #0 fix: add unique relation for source config table (Sanketika-Obsrv#110)

* feat new image tags (Sanketika-Obsrv#112)

* * #0000 - feat: Fix masterdata stats ingestion spec
* #0000 - feat: Add latency and processing times

* * #0000 - feat: Flink helm configuration changes

* #0000 - feat: Ingestion spec changes for system-events

* #0000 - feat: Remove outdated ingestion specs

* #000: Rebased the develop with release-1.0.0-GA

* #0000 - feat: Import error dashboard (Sanketika-Obsrv#119)

* [ Secor Timzone ] - Infra automation changes to take the timezone property (Sanketika-Obsrv#120)

* #000: Configuring the helm to handle the instance level timezone

* Removed the unwanted timezone properties

* Sanketika-Obsrv#203: Removed the commented code

* Sanketika-Obsrv#226 Feat: Update postgres migrations to add system settings and default values (Sanketika-Obsrv#121)

* Sanketika-Obsrv#226 fix: add system_settings table

* Sanketika-Obsrv#226 fix: Add data encryption key to PostgreSQL migration

* Sanketika-Obsrv#226 fix: move system settings new migration script

* update tags for ga release and fix undeclared variable in secor (Sanketika-Obsrv#122)

* System Telemetry Events Ingestion Spec and Backup Configurations (Sanketika-Obsrv#118)

* Sanketika-Obsrv#223: Configured the auto submission of telemetry events ingestion spec and configured the secor backup for system telemetry events

* Sanketika-Obsrv#223: Updated the secor jobs variabl by including system-telemetry-events process

* Sanketika-Obsrv#203: typo error fix

* Sanketika-Obsrv#203: Configured the flag to submit the ingestion spec

* Sanketika-Obsrv#203: Configured the flag to submit the ingestion spec

* Sanketika-Obsrv#67 fix: add grant permissions for created table (Sanketika-Obsrv#123)

* Sanketika-Obsrv#172 feat: add outputs for kafka service (Sanketika-Obsrv#130)

* Sanketika-Obsrv#172 feat: add outputs for kafka service (Sanketika-Obsrv#132)

* Sanketika-Obsrv#245 feat: Update Druid, PostgreSQL, and Superset versions (Sanketika-Obsrv#133)

* #0 feat: Update Druid, PostgreSQL, and Superset versions

* Sanketika-Obsrv#245 fix: Grant all privileges on schema to superset and druid_raw

* Sanketika-Obsrv#245 fix: Update database ownership in migration scripts

* Sanketika-Obsrv#241 Enhance Infrastructure Flexibility, Deployment Reliability, and Observability (Sanketika-Obsrv#131)

* Sanketika-Obsrv#241:feat: Infra automation script

* Sanketika-Obsrv#241 feat: Toggle vpc creation, Toggle velero user creation

* update: Sanketika-Obsrv#241 main.tf for velero user creation

* remove: Sanketika-Obsrv#241 revert automation script changes

* update: Sanketika-Obsrv#241 Toggle to change service type and dataset api cm bug fix

* Sanketika-Obsrv#241:feat: Updated the infra setup shell script

* Sanketika-Obsrv#241:doc: Updated the setup doc file

* Sanketika-Obsrv#241:doc: Updated the setup doc file and removed commented code

* Sanketika-Obsrv#241:doc: Added the Prerequisites section

* feat: Sanketika-Obsrv#241 Toggle to enable cloudwatch logs for EKS

* feat: Sanketika-Obsrv#241 Deployment changes to provision infra

* feat: # 241 Deployment script for provisioning infra

* clean: Sanketika-Obsrv#241 remove unwanted code.

* update: Sanketika-Obsrv#241 update helm version in shell script and remove overrides.tfvars file

* enhance: Sanketika-Obsrv#241 Enhance shell script create a tfvars from setup.conf file

* fix: Sanketika-Obsrv#241 typo in displayed message, fix terraform installation command

* update: Sanketika-Obsrv#241 eks cluster version

* update: Sanketika-Obsrv#241 kuberentes service type

* update: Sanketika-Obsrv#241 kubernetes service type to loadbalancer

* Update helm version

---------



* update: AKS cluster version (Sanketika-Obsrv#134)

* bug fix: Sanketika-Obsrv#241 region typo and modify shell script (Sanketika-Obsrv#135)

* bug fix: Sanketika-Obsrv#241 region typo and modify shell script

* Bug fix: Sanketika-Obsrv#241 modify velero module

* Installation fixes (Sanketika-Obsrv#138)

* fix: installation-fixes and enhancements

* update: [1/2] Enabling cluster logs by default. [2/2] Creating vpc, kong ingress, velero user by default

* feat: Enable kafka connector and fix class name for kafka connector

* update: add decrypt command

* upgrade: eks version

* dummy commit to reflect changes

* dummy commit to reflect changes

* modify desired node capacity

* revert node capacity changes

---------

Co-authored-by: GayathriSrividya <[email protected]>
Co-authored-by: divyagovindaiah <[email protected]>
Co-authored-by: Praveen <[email protected]>
Co-authored-by: Manoj Krishna <[email protected]>
Co-authored-by: Sowmya N Dixit <[email protected]>
Co-authored-by: Ravi Mula <[email protected]>
Co-authored-by: Ravi Mula <[email protected]>
Co-authored-by: Anand Parthasarathy <[email protected]>
Co-authored-by: Ravinder Kumar <[email protected]>

automation-scripts/infra-setup/setup.conf

@@ -17,7 +17,7 @@ BUILDING_BLOCK="" # obsrv
 ENV="" # dev
 REGION="" # eu-west-1
 TIMEZONE="UTC"
-ALLOW_VPC_CREATION=false # It will creates VPC, Subnets.
-ALLOW_VELERO_USER_CREATION=false # It will create velero user
-ALLOW_KONG_INGRESS_SETUP=false # It will setup kong ingress
+ALLOW_VPC_CREATION=true # It will creates VPC, Subnets.
+ALLOW_VELERO_USER_CREATION=true # It will create velero user
+ALLOW_KONG_INGRESS_SETUP=true # It will setup kong ingress
 # Add more variables as needed

automation-scripts/infra-setup/setup.sh

@@ -179,6 +179,7 @@ validate_tools
 
 # Script related to terraform and deployment will start from here
 cd ../../terraform/aws
+terrahelp decrypt  -simple-key=<decryption_key> -file=vars/dev.tfvars
 terragrunt init
 terragrunt apply -target module.eks -var "create_vpc=$ALLOW_VPC_CREATION" -var "create_velero_user=$ALLOW_VELERO_USER_CREATION"  -var-file=vars/dev.tfvars -var-file=vars/overrides.tfvars -auto-approve
 terragrunt apply -target module.get_kubeconfig -var "create_vpc=$ALLOW_VPC_CREATION" -var "create_velero_user=$ALLOW_VELERO_USER_CREATION"  -var-file=vars/dev.tfvars -var-file=vars/overrides.tfvars -auto-approve

terraform/aws/main.tf

@@ -79,6 +79,7 @@ module "monitoring" {
   source                           = "../modules/helm/monitoring"
   env                              = var.env
   building_block                   = var.building_block
+  service_type                     = var.service_type
   depends_on                       = [module.eks]
 }
 

terraform/aws/variables.tf

@@ -54,12 +54,13 @@ variable "velero_aws_secret_access_key" {
 }
 variable "service_type" {
   type = string
-  description = "Kubernetes service type either NodePort or LoadBalancer. It is NodePort by default"
+  description = "Kubernetes service type either NodePort or LoadBalancer. It is LoadBalancer by default"
   default = "LoadBalancer"
 }
 variable "cluster_logs_enabled" {
   type = bool
   description = "Toggle to enable eks cluster logs"
+  default = true
 }
 variable "flink_checkpoint_store_type" {
   type        = string
@@ -140,7 +141,7 @@ variable "flink_release_names" {
     transformer     = "transformer"
     druid-router    = "druid-router"
     master-data-processor = "master-data-processor"
-    # kafka-connector = "kafka-connector"
+    kafka-connector = "kafka-connector"
   }
 }
 
@@ -150,7 +151,7 @@ variable "flink_merged_pipeline_release_names" {
   default = {
     merged-pipeline = "merged-pipeline"
     master-data-processor = "master-data-processor"
-    # kafka-connector = "kafka-connector"
+    kafka-connector = "kafka-connector"
   }
 }
 

terraform/modules/aws/eks/variables.tf

@@ -77,7 +77,7 @@ variable "eks_node_group_scaling_config" {
 variable "eks_version" {
   type        = string
   description = "EKS version."
-  default     = "1.27"
+  default     = "1.28"
 }
 
 variable "eks_addons" {

terraform/modules/helm/flink/flink-helm-chart/values.yaml

@@ -419,7 +419,7 @@ kafka-connector:
     heartbeat.interval: 5000
     state.savepoints.dir: file:///tmp
 
-  job_classname: org.sunbird.obsrv.kafkaconnector.task.KafkaConnectorStreamTask
+  job_classname: org.sunbird.obsrv.connector.task.KafkaConnectorStreamTask
 
 master-data-processor:
   master-data-processor: |+

terraform/modules/helm/monitoring/main.tf

@@ -15,6 +15,7 @@ resource "helm_release" "monitoring" {
         {
             prometheus_persistent_volume_size = var.prometheus_persistent_volume_size
             prometheus_metrics_retention      = var.prometheus_metrics_retention
+            service_type                      = var.service_type
         }
       )
     ]

terraform/modules/helm/monitoring/monitoring.yaml.tfpl

@@ -32,6 +32,9 @@ kube-state-metrics:
       cpu: "0.1"
       memory: 128Mi
 grafana:
+  service:
+    enabled: true
+    type: ${service_type}
   resources:
     limits:
       cpu: 0.2

terraform/modules/helm/monitoring/variables.tf

@@ -7,6 +7,10 @@ variable "building_block" {
   type        = string
   description = "Building block name. All resources will be prefixed with this value."
 }
+variable "service_type" {
+  type = string
+  default = "Kubernetes service type either LoadBalancer or NodePort. It is LoadBalancer by default"
+}
 
 variable "monitoring_release_name" {
   type        = string