-
Notifications
You must be signed in to change notification settings - Fork 108
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* optimize kpi * optimize kube node inventory * add flags for events, deployments and hpa * have separate function parseNodeLimits * refactor code * fix crash * fix bug with service name * fix bugs related to get service name * update oom fix test agent * debug logs * fix service label issue * update to latest agent and enable ephemeral annotation * change stream size to 200 from 250 * update yaml * adjust chunksizes * add ruby gc env * yaml changes for cioomtest11282020-3 * telemetry to track pods latency * service count telemetry * rename variables * wip * nodes inventory telemetry * configmap changes * add emit streams in configmap * yaml updates * fix copy and paste bug * add todo comments * fix node latency telemetry bug * update yaml with latest test image * fix bug * upping rs memory change * fix mdm bug with final emit stream * update to latest image * fix pr feedback * fix pr feedback * rename health config to agent config * fix max allowed hpa chunk size * update to use 1k pod chunk since validated on 1.18+ * remove debug logs * minor updates * move defaults to common place * chart updates * final oomfix agent * update to use prod image so that can be validated with build pipeline * fix typo in comment
- Loading branch information
Showing
14 changed files
with
1,534 additions
and
1,166 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
172 changes: 172 additions & 0 deletions
172
build/linux/installer/scripts/tomlparser-agent-config.rb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,172 @@ | ||
| #!/usr/local/bin/ruby | ||
|
|
||
| #this should be require relative in Linux and require in windows, since it is a gem install on windows | ||
| @os_type = ENV["OS_TYPE"] | ||
| if !@os_type.nil? && !@os_type.empty? && @os_type.strip.casecmp("windows") == 0 | ||
| require "tomlrb" | ||
| else | ||
| require_relative "tomlrb" | ||
| end | ||
|
|
||
| require_relative "ConfigParseErrorLogger" | ||
|
|
||
| @configMapMountPath = "/etc/config/settings/agent-settings" | ||
| @configSchemaVersion = "" | ||
| @enable_health_model = false | ||
|
|
||
| # 250 Node items (15KB per node) account to approximately 4MB | ||
| @nodesChunkSize = 250 | ||
| # 1000 pods (10KB per pod) account to approximately 10MB | ||
| @podsChunkSize = 1000 | ||
| # 4000 events (1KB per event) account to approximately 4MB | ||
| @eventsChunkSize = 4000 | ||
| # roughly each deployment is 8k | ||
| # 500 deployments account to approximately 4MB | ||
| @deploymentsChunkSize = 500 | ||
| # roughly each HPA is 3k | ||
| # 2000 HPAs account to approximately 6-7MB | ||
| @hpaChunkSize = 2000 | ||
| # stream batch sizes to avoid large file writes | ||
| # too low will consume higher disk iops | ||
| @podsEmitStreamBatchSize = 200 | ||
| @nodesEmitStreamBatchSize = 100 | ||
|
|
||
| # higher the chunk size rs pod memory consumption higher and lower api latency | ||
| # similarly lower the value, helps on the memory consumption but incurrs additional round trip latency | ||
| # these needs to be tuned be based on the workload | ||
| # nodes | ||
| @nodesChunkSizeMin = 100 | ||
| @nodesChunkSizeMax = 400 | ||
| # pods | ||
| @podsChunkSizeMin = 250 | ||
| @podsChunkSizeMax = 1500 | ||
| # events | ||
| @eventsChunkSizeMin = 2000 | ||
| @eventsChunkSizeMax = 10000 | ||
| # deployments | ||
| @deploymentsChunkSizeMin = 500 | ||
| @deploymentsChunkSizeMax = 1000 | ||
| # hpa | ||
| @hpaChunkSizeMin = 500 | ||
| @hpaChunkSizeMax = 2000 | ||
|
|
||
| # emit stream sizes to prevent lower values which costs disk i/o | ||
| # max will be upto the chunk size | ||
| @podsEmitStreamBatchSizeMin = 50 | ||
| @nodesEmitStreamBatchSizeMin = 50 | ||
|
|
||
| def is_number?(value) | ||
| true if Integer(value) rescue false | ||
| end | ||
|
|
||
| # Use parser to parse the configmap toml file to a ruby structure | ||
| def parseConfigMap | ||
| begin | ||
| # Check to see if config map is created | ||
| if (File.file?(@configMapMountPath)) | ||
| puts "config::configmap container-azm-ms-agentconfig for agent settings mounted, parsing values" | ||
| parsedConfig = Tomlrb.load_file(@configMapMountPath, symbolize_keys: true) | ||
| puts "config::Successfully parsed mounted config map" | ||
| return parsedConfig | ||
| else | ||
| puts "config::configmap container-azm-ms-agentconfig for agent settings not mounted, using defaults" | ||
| return nil | ||
| end | ||
| rescue => errorStr | ||
| ConfigParseErrorLogger.logError("Exception while parsing config map for agent settings : #{errorStr}, using defaults, please check config map for errors") | ||
| return nil | ||
| end | ||
| end | ||
|
|
||
| # Use the ruby structure created after config parsing to set the right values to be used as environment variables | ||
| def populateSettingValuesFromConfigMap(parsedConfig) | ||
| begin | ||
| if !parsedConfig.nil? && !parsedConfig[:agent_settings].nil? | ||
| if !parsedConfig[:agent_settings][:health_model].nil? && !parsedConfig[:agent_settings][:health_model][:enabled].nil? | ||
| @enable_health_model = parsedConfig[:agent_settings][:health_model][:enabled] | ||
| puts "enable_health_model = #{@enable_health_model}" | ||
| end | ||
| chunk_config = parsedConfig[:agent_settings][:chunk_config] | ||
| if !chunk_config.nil? | ||
| nodesChunkSize = chunk_config[:NODES_CHUNK_SIZE] | ||
| if !nodesChunkSize.nil? && is_number?(nodesChunkSize) && (@nodesChunkSizeMin..@nodesChunkSizeMax) === nodesChunkSize.to_i | ||
| @nodesChunkSize = nodesChunkSize.to_i | ||
| puts "Using config map value: NODES_CHUNK_SIZE = #{@nodesChunkSize}" | ||
| end | ||
|
|
||
| podsChunkSize = chunk_config[:PODS_CHUNK_SIZE] | ||
| if !podsChunkSize.nil? && is_number?(podsChunkSize) && (@podsChunkSizeMin..@podsChunkSizeMax) === podsChunkSize.to_i | ||
| @podsChunkSize = podsChunkSize.to_i | ||
| puts "Using config map value: PODS_CHUNK_SIZE = #{@podsChunkSize}" | ||
| end | ||
|
|
||
| eventsChunkSize = chunk_config[:EVENTS_CHUNK_SIZE] | ||
| if !eventsChunkSize.nil? && is_number?(eventsChunkSize) && (@eventsChunkSizeMin..@eventsChunkSizeMax) === eventsChunkSize.to_i | ||
| @eventsChunkSize = eventsChunkSize.to_i | ||
| puts "Using config map value: EVENTS_CHUNK_SIZE = #{@eventsChunkSize}" | ||
| end | ||
|
|
||
| deploymentsChunkSize = chunk_config[:DEPLOYMENTS_CHUNK_SIZE] | ||
| if !deploymentsChunkSize.nil? && is_number?(deploymentsChunkSize) && (@deploymentsChunkSizeMin..@deploymentsChunkSizeMax) === deploymentsChunkSize.to_i | ||
| @deploymentsChunkSize = deploymentsChunkSize.to_i | ||
| puts "Using config map value: DEPLOYMENTS_CHUNK_SIZE = #{@deploymentsChunkSize}" | ||
| end | ||
|
|
||
| hpaChunkSize = chunk_config[:HPA_CHUNK_SIZE] | ||
| if !hpaChunkSize.nil? && is_number?(hpaChunkSize) && (@hpaChunkSizeMin..@hpaChunkSizeMax) === hpaChunkSize.to_i | ||
| @hpaChunkSize = hpaChunkSize.to_i | ||
| puts "Using config map value: HPA_CHUNK_SIZE = #{@hpaChunkSize}" | ||
| end | ||
|
|
||
| podsEmitStreamBatchSize = chunk_config[:PODS_EMIT_STREAM_BATCH_SIZE] | ||
| if !podsEmitStreamBatchSize.nil? && is_number?(podsEmitStreamBatchSize) && | ||
| podsEmitStreamBatchSize.to_i <= @podsChunkSize && podsEmitStreamBatchSize.to_i >= @podsEmitStreamBatchSizeMin | ||
| @podsEmitStreamBatchSize = podsEmitStreamBatchSize.to_i | ||
| puts "Using config map value: PODS_EMIT_STREAM_BATCH_SIZE = #{@podsEmitStreamBatchSize}" | ||
| end | ||
| nodesEmitStreamBatchSize = chunk_config[:NODES_EMIT_STREAM_BATCH_SIZE] | ||
| if !nodesEmitStreamBatchSize.nil? && is_number?(nodesEmitStreamBatchSize) && | ||
| nodesEmitStreamBatchSize.to_i <= @nodesChunkSize && nodesEmitStreamBatchSize.to_i >= @nodesEmitStreamBatchSizeMin | ||
| @nodesEmitStreamBatchSize = nodesEmitStreamBatchSize.to_i | ||
| puts "Using config map value: NODES_EMIT_STREAM_BATCH_SIZE = #{@nodesEmitStreamBatchSize}" | ||
| end | ||
| end | ||
| end | ||
| rescue => errorStr | ||
| puts "config::error:Exception while reading config settings for agent configuration setting - #{errorStr}, using defaults" | ||
| @enable_health_model = false | ||
| end | ||
| end | ||
|
|
||
| @configSchemaVersion = ENV["AZMON_AGENT_CFG_SCHEMA_VERSION"] | ||
| puts "****************Start Config Processing********************" | ||
| if !@configSchemaVersion.nil? && !@configSchemaVersion.empty? && @configSchemaVersion.strip.casecmp("v1") == 0 #note v1 is the only supported schema version , so hardcoding it | ||
| configMapSettings = parseConfigMap | ||
| if !configMapSettings.nil? | ||
| populateSettingValuesFromConfigMap(configMapSettings) | ||
| end | ||
| else | ||
| if (File.file?(@configMapMountPath)) | ||
| ConfigParseErrorLogger.logError("config::unsupported/missing config schema version - '#{@configSchemaVersion}' , using defaults, please use supported schema version") | ||
| end | ||
| @enable_health_model = false | ||
| end | ||
|
|
||
| # Write the settings to file, so that they can be set as environment variables | ||
| file = File.open("agent_config_env_var", "w") | ||
|
|
||
| if !file.nil? | ||
| file.write("export AZMON_CLUSTER_ENABLE_HEALTH_MODEL=#{@enable_health_model}\n") | ||
| file.write("export NODES_CHUNK_SIZE=#{@nodesChunkSize}\n") | ||
| file.write("export PODS_CHUNK_SIZE=#{@podsChunkSize}\n") | ||
| file.write("export EVENTS_CHUNK_SIZE=#{@eventsChunkSize}\n") | ||
| file.write("export DEPLOYMENTS_CHUNK_SIZE=#{@deploymentsChunkSize}\n") | ||
| file.write("export HPA_CHUNK_SIZE=#{@hpaChunkSize}\n") | ||
| file.write("export PODS_EMIT_STREAM_BATCH_SIZE=#{@podsEmitStreamBatchSize}\n") | ||
| file.write("export NODES_EMIT_STREAM_BATCH_SIZE=#{@nodesEmitStreamBatchSize}\n") | ||
| # Close file after writing all environment variables | ||
| file.close | ||
| else | ||
| puts "Exception while opening file for writing config environment variables" | ||
| puts "****************End Config Processing********************" | ||
| end |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.