Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prometheus plugin - series with +Inf , -Inf, NaN values fail for json output as its not a valid json for value #7388

Closed
vishiy opened this issue Apr 22, 2020 · 0 comments · Fixed by #7426
Labels
bug unexpected problem or unintended behavior
Milestone

Comments

@vishiy
Copy link
Contributor

vishiy commented Apr 22, 2020

Relevant telegraf.conf:

###############################################################################
#                            OUTPUT PLUGINS                                   #
###############################################################################

# Generic socket writer capable of handling multiple socket types.
[[outputs.socket_writer]]
  ## URL to connect to
  address = "tcp://0.0.0.0:25226"
  

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

  ## Period between keep alive probes.
  ## Only applies to TCP sockets.
  ## 0 disables keep alive probes.
  ## Defaults to the OS configuration.
  # keep_alive_period = "5m"

  ## Data format to generate.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "json"

[[inputs.prometheus]]
  interval = "1m"
  
  ## Scrape Kubernetes pods for the following prometheus annotations:
  ## - prometheus.io/scrape: Enable scraping for this pod
  ## - prometheus.io/scheme: If the metrics endpoint is secured then you will need to
  ##     set this to `https` & most likely set the tls config.
  ## - prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.
  ## - prometheus.io/port: If port is not 9102 use this annotation
  monitor_kubernetes_pods = true
  metric_version = 2
  ## Kubernetes config file to create client from.
  # kube_config = "/path/to/kubernetes.config"
  ## Use bearer token for authorization. ('bearer_token' takes priority)
  bearer_token = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  ## OR
  # bearer_token_string = "abc_123"
  ## Specify timeout duration for slower prometheus clients (default is 3s)
  response_timeout = "15s"

  ## Optional TLS Config
  tls_ca = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
  #tls_cert = /path/to/certfile
  # tls_key = /path/to/keyfile
  ## Use TLS but skip chain & host verification
  insecure_skip_verify = true

System info:

Docker

Steps to reproduce:

  1. Here is a gauge metric with -ve infinity as value (-Inf) - This is a correct value as per prometheus line format (+Inf , -Inf & NaN are valid values)

TYPE flink_taskmanager_job_task_operator_KafkaProducer_record_size_max gauge

flink_taskmanager_job_task_operator_KafkaProducer_record_size_max{job_id="40c668ac8435480261a4c2afb8f88deb",task_id="c8e26e1de46a579b823e994ab7ebd862",task_attempt_id="f099f1bb3403a8762dfc12af3a580c29",host="flink_tm_0",operator_id="fd06000fa5e590dc6d2474670a0b4c2c",operator_name="Sink:_Unnamed",task_name="KeyedProcess_____Sink:_Unnamed__Sink:Unnamed",task_attempt_num="0",job_name="pulse_TictracService",tm_id="7f41cff8bb3199807ca9b9bb164eed23",subtask_index="2",} -Inf

  1. Stitch the prometheus input with json output format, and you will see the below error -
    2020-04-21T01:01:05Z E! [agent] Error writing to output [socket_writer]: json: unsupported value: -Inf

Expected behavior:

No error when serializing as JSON
Looks to me we are parsing values using float64() rather than parseFloat(), which is the root cause.

@vishiy vishiy changed the title Prometheus plugin - +Inf & -Inf values fail for json output as its not a valid json for value Prometheus plugin - series with +Inf , -Inf, NaN values fail for json output as its not a valid json for value Apr 22, 2020
@danielnelson danielnelson added this to the 1.14.2 milestone Apr 22, 2020
@danielnelson danielnelson added the bug unexpected problem or unintended behavior label Apr 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug unexpected problem or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants