From e1d7890de635962177ca40edabd4e773406db8d5 Mon Sep 17 00:00:00 2001 From: Manjunath Davanam Date: Mon, 29 Jan 2024 14:52:50 +0530 Subject: [PATCH] Develop to 1.0.2-GA (#140) (#141) * Fix: removed docker secrets * updated the grafana_secrete * Feat: updated image tags according to release 1.1.0 * Issue #171 feat: update job to deployment for jobmanagers * #90 fix: Add default value for published_date in tables (#94) * fix: remove restart policy from flink deployment manifest (#96) * #196 Fix:Remove default region. (#95) * fix: #196 remove default value for region * update: #196 missing variable in variables.tf file * #196 update: example on how provide region * #196 update: comments on how to provide region * Separation of redis into dedup & meta/denorm. (#92) * feat: Separation of redis into dedup & meta/denorm. * feat: Separation of redis into dedup & meta/denorm. * feat: Separation of redis into dedup & meta/denorm. * feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * develop into release 1.3.0 (#97) * #90 fix: Add default value for published_date in tables (#94) * fix: remove restart policy from flink deployment manifest (#96) * #196 Fix:Remove default region. (#95) * fix: #196 remove default value for region * update: #196 missing variable in variables.tf file * #196 update: example on how provide region * #196 update: comments on how to provide region * Separation of redis into dedup & meta/denorm. (#92) * feat: Separation of redis into dedup & meta/denorm. * feat: Separation of redis into dedup & meta/denorm. * feat: Separation of redis into dedup & meta/denorm. * feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. * #58 feat: Separation of redis into dedup & meta/denorm. --------- * #126 feat: add new property to datasources table (#99) * #126 feat: add new property to datasources table * #126 fix: allow property to be nullable * #0000 - feat: Fix masterdata stats ingestion spec (#100) * #0000 - feat: Fix masterdata stats ingestion spec * #0000 - feat: Add latency and processing times * #0000 - feat: Flink helm configuration changes (#103) * #0000 - feat: Flink helm configuration changes * #0000 - feat: Ingestion spec changes for system-events * #0000 - feat: Remove outdated ingestion specs * #0000 - feat: Secor changes for unified failed topic (#104) * feat: azure terraform changes to provision cluster (#106) * feat: azure terraform changes to provision cluster * cleanup azure terraform code * update node count and instance size for azure (#107) * Issue #305 feat: exclude system-events for the datasource validation (#108) * Issue #0000 fix: system events ingestion spec typo fix (#105) * Issue #0000 fix: system events ingestion spec typo fix * Issue #0000 fix: system-events ingestion spec update to add dataset_type * #0 fix: add unique relation for source config table (#110) * feat new image tags (#112) * * #0000 - feat: Fix masterdata stats ingestion spec * #0000 - feat: Add latency and processing times * * #0000 - feat: Flink helm configuration changes * #0000 - feat: Ingestion spec changes for system-events * #0000 - feat: Remove outdated ingestion specs * #000: Rebased the develop with release-1.0.0-GA * #0000 - feat: Import error dashboard (#119) * [ Secor Timzone ] - Infra automation changes to take the timezone property (#120) * #000: Configuring the helm to handle the instance level timezone * Removed the unwanted timezone properties * #203: Removed the commented code * #226 Feat: Update postgres migrations to add system settings and default values (#121) * #226 fix: add system_settings table * #226 fix: Add data encryption key to PostgreSQL migration * #226 fix: move system settings new migration script * update tags for ga release and fix undeclared variable in secor (#122) * System Telemetry Events Ingestion Spec and Backup Configurations (#118) * #223: Configured the auto submission of telemetry events ingestion spec and configured the secor backup for system telemetry events * #223: Updated the secor jobs variabl by including system-telemetry-events process * #203: typo error fix * #203: Configured the flag to submit the ingestion spec * #203: Configured the flag to submit the ingestion spec * #67 fix: add grant permissions for created table (#123) * #172 feat: add outputs for kafka service (#130) * #172 feat: add outputs for kafka service (#132) * #245 feat: Update Druid, PostgreSQL, and Superset versions (#133) * #0 feat: Update Druid, PostgreSQL, and Superset versions * #245 fix: Grant all privileges on schema to superset and druid_raw * #245 fix: Update database ownership in migration scripts * #241 Enhance Infrastructure Flexibility, Deployment Reliability, and Observability (#131) * #241:feat: Infra automation script * #241 feat: Toggle vpc creation, Toggle velero user creation * update: #241 main.tf for velero user creation * remove: #241 revert automation script changes * update: #241 Toggle to change service type and dataset api cm bug fix * #241:feat: Updated the infra setup shell script * #241:doc: Updated the setup doc file * #241:doc: Updated the setup doc file and removed commented code * #241:doc: Added the Prerequisites section * feat: #241 Toggle to enable cloudwatch logs for EKS * feat: #241 Deployment changes to provision infra * feat: # 241 Deployment script for provisioning infra * clean: #241 remove unwanted code. * update: #241 update helm version in shell script and remove overrides.tfvars file * enhance: #241 Enhance shell script create a tfvars from setup.conf file * fix: #241 typo in displayed message, fix terraform installation command * update: #241 eks cluster version * update: #241 kuberentes service type * update: #241 kubernetes service type to loadbalancer * Update helm version --------- * update: AKS cluster version (#134) * bug fix: #241 region typo and modify shell script (#135) * bug fix: #241 region typo and modify shell script * Bug fix: #241 modify velero module * Installation fixes (#138) * fix: installation-fixes and enhancements * update: [1/2] Enabling cluster logs by default. [2/2] Creating vpc, kong ingress, velero user by default * feat: Enable kafka connector and fix class name for kafka connector * update: add decrypt command * upgrade: eks version * dummy commit to reflect changes * dummy commit to reflect changes * modify desired node capacity * revert node capacity changes --------- Co-authored-by: GayathriSrividya Co-authored-by: divyagovindaiah Co-authored-by: Praveen <66662436+pveleneni@users.noreply.github.com> Co-authored-by: Manoj Krishna <92361832+ManojKrishnaChintaluri@users.noreply.github.com> Co-authored-by: Sowmya N Dixit Co-authored-by: Ravi Mula Co-authored-by: Ravi Mula Co-authored-by: Anand Parthasarathy Co-authored-by: Ravinder Kumar --- automation-scripts/infra-setup/setup.conf | 6 +++--- automation-scripts/infra-setup/setup.sh | 1 + terraform/aws/main.tf | 1 + terraform/aws/variables.tf | 7 ++++--- terraform/modules/aws/eks/variables.tf | 2 +- terraform/modules/helm/flink/flink-helm-chart/values.yaml | 2 +- terraform/modules/helm/monitoring/main.tf | 1 + terraform/modules/helm/monitoring/monitoring.yaml.tfpl | 3 +++ terraform/modules/helm/monitoring/variables.tf | 4 ++++ 9 files changed, 19 insertions(+), 8 deletions(-) diff --git a/automation-scripts/infra-setup/setup.conf b/automation-scripts/infra-setup/setup.conf index df3df284..bc047e04 100644 --- a/automation-scripts/infra-setup/setup.conf +++ b/automation-scripts/infra-setup/setup.conf @@ -17,7 +17,7 @@ BUILDING_BLOCK="" # obsrv ENV="" # dev REGION="" # eu-west-1 TIMEZONE="UTC" -ALLOW_VPC_CREATION=false # It will creates VPC, Subnets. -ALLOW_VELERO_USER_CREATION=false # It will create velero user -ALLOW_KONG_INGRESS_SETUP=false # It will setup kong ingress +ALLOW_VPC_CREATION=true # It will creates VPC, Subnets. +ALLOW_VELERO_USER_CREATION=true # It will create velero user +ALLOW_KONG_INGRESS_SETUP=true # It will setup kong ingress # Add more variables as needed \ No newline at end of file diff --git a/automation-scripts/infra-setup/setup.sh b/automation-scripts/infra-setup/setup.sh index 1bbb4dc4..fa2c256c 100644 --- a/automation-scripts/infra-setup/setup.sh +++ b/automation-scripts/infra-setup/setup.sh @@ -179,6 +179,7 @@ validate_tools # Script related to terraform and deployment will start from here cd ../../terraform/aws +terrahelp decrypt -simple-key= -file=vars/dev.tfvars terragrunt init terragrunt apply -target module.eks -var "create_vpc=$ALLOW_VPC_CREATION" -var "create_velero_user=$ALLOW_VELERO_USER_CREATION" -var-file=vars/dev.tfvars -var-file=vars/overrides.tfvars -auto-approve terragrunt apply -target module.get_kubeconfig -var "create_vpc=$ALLOW_VPC_CREATION" -var "create_velero_user=$ALLOW_VELERO_USER_CREATION" -var-file=vars/dev.tfvars -var-file=vars/overrides.tfvars -auto-approve diff --git a/terraform/aws/main.tf b/terraform/aws/main.tf index ca43ea32..c9a180a7 100644 --- a/terraform/aws/main.tf +++ b/terraform/aws/main.tf @@ -79,6 +79,7 @@ module "monitoring" { source = "../modules/helm/monitoring" env = var.env building_block = var.building_block + service_type = var.service_type depends_on = [module.eks] } diff --git a/terraform/aws/variables.tf b/terraform/aws/variables.tf index 3129d9a3..282b8f5f 100644 --- a/terraform/aws/variables.tf +++ b/terraform/aws/variables.tf @@ -54,12 +54,13 @@ variable "velero_aws_secret_access_key" { } variable "service_type" { type = string - description = "Kubernetes service type either NodePort or LoadBalancer. It is NodePort by default" + description = "Kubernetes service type either NodePort or LoadBalancer. It is LoadBalancer by default" default = "LoadBalancer" } variable "cluster_logs_enabled" { type = bool description = "Toggle to enable eks cluster logs" + default = true } variable "flink_checkpoint_store_type" { type = string @@ -140,7 +141,7 @@ variable "flink_release_names" { transformer = "transformer" druid-router = "druid-router" master-data-processor = "master-data-processor" - # kafka-connector = "kafka-connector" + kafka-connector = "kafka-connector" } } @@ -150,7 +151,7 @@ variable "flink_merged_pipeline_release_names" { default = { merged-pipeline = "merged-pipeline" master-data-processor = "master-data-processor" - # kafka-connector = "kafka-connector" + kafka-connector = "kafka-connector" } } diff --git a/terraform/modules/aws/eks/variables.tf b/terraform/modules/aws/eks/variables.tf index b3b99d74..d645e0a1 100644 --- a/terraform/modules/aws/eks/variables.tf +++ b/terraform/modules/aws/eks/variables.tf @@ -77,7 +77,7 @@ variable "eks_node_group_scaling_config" { variable "eks_version" { type = string description = "EKS version." - default = "1.27" + default = "1.28" } variable "eks_addons" { diff --git a/terraform/modules/helm/flink/flink-helm-chart/values.yaml b/terraform/modules/helm/flink/flink-helm-chart/values.yaml index f9f2c86b..84ae15ef 100644 --- a/terraform/modules/helm/flink/flink-helm-chart/values.yaml +++ b/terraform/modules/helm/flink/flink-helm-chart/values.yaml @@ -419,7 +419,7 @@ kafka-connector: heartbeat.interval: 5000 state.savepoints.dir: file:///tmp - job_classname: org.sunbird.obsrv.kafkaconnector.task.KafkaConnectorStreamTask + job_classname: org.sunbird.obsrv.connector.task.KafkaConnectorStreamTask master-data-processor: master-data-processor: |+ diff --git a/terraform/modules/helm/monitoring/main.tf b/terraform/modules/helm/monitoring/main.tf index 607677a8..66517097 100644 --- a/terraform/modules/helm/monitoring/main.tf +++ b/terraform/modules/helm/monitoring/main.tf @@ -15,6 +15,7 @@ resource "helm_release" "monitoring" { { prometheus_persistent_volume_size = var.prometheus_persistent_volume_size prometheus_metrics_retention = var.prometheus_metrics_retention + service_type = var.service_type } ) ] diff --git a/terraform/modules/helm/monitoring/monitoring.yaml.tfpl b/terraform/modules/helm/monitoring/monitoring.yaml.tfpl index 73fa3053..ec68a56c 100644 --- a/terraform/modules/helm/monitoring/monitoring.yaml.tfpl +++ b/terraform/modules/helm/monitoring/monitoring.yaml.tfpl @@ -32,6 +32,9 @@ kube-state-metrics: cpu: "0.1" memory: 128Mi grafana: + service: + enabled: true + type: ${service_type} resources: limits: cpu: 0.2 diff --git a/terraform/modules/helm/monitoring/variables.tf b/terraform/modules/helm/monitoring/variables.tf index 5cbe2471..2c4b77b6 100644 --- a/terraform/modules/helm/monitoring/variables.tf +++ b/terraform/modules/helm/monitoring/variables.tf @@ -7,6 +7,10 @@ variable "building_block" { type = string description = "Building block name. All resources will be prefixed with this value." } +variable "service_type" { + type = string + default = "Kubernetes service type either LoadBalancer or NodePort. It is LoadBalancer by default" +} variable "monitoring_release_name" { type = string