Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Envoy check crashes when collecting metrics from Istio Envoy #7686

Closed
max0ne opened this issue Sep 30, 2020 · 1 comment · Fixed by #7717
Closed

Envoy check crashes when collecting metrics from Istio Envoy #7686

max0ne opened this issue Sep 30, 2020 · 1 comment · Fixed by #7717
Assignees
@ofek

Comments

@max0ne
Copy link

max0ne commented Sep 30, 2020

Output of the info page

Getting the status from the agent.

===============
Agent (v6.20.0)
===============

  Status date: 2020-09-30 03:44:21.621801 UTC
  Agent start: 2020-09-30 03:41:33.917383 UTC
  Pid: 347
  Go Version: go1.13.8
  Python Version: 2.7.18
  Build arch: amd64
  Check Runners: 4
  Log Level: warn

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: 398µs
    System UTC time: 2020-09-30 03:44:21.621801 UTC

  Host Info
  =========
    bootTime: 2020-09-29 00:26:00.000000 UTC
    kernelArch: x86_64
    kernelVersion: 5.0.0-1033-gcp
    os: linux
    platform: debian
    platformFamily: debian
    platformVersion: bullseye/sid
    procs: 10
    uptime: 27h15m36s
    virtualizationRole: guest
    virtualizationSystem: docker

  Hostnames
  =========
    host_aliases: <redacted>
    hostname: <redacted>
    socket-fqdn: <redacted>
    socket-hostname: <redacted>
    host tags:
      datadog_version:6.20.0
      <many tags redacted>
    hostname provider: gce
    unused hostname providers:
      configuration/environment: hostname is empty

  Metadata
  ========
    cloud_provider: GCP
    hostname_source: gce

=========
Collector
=========

  Running Checks
  ==============

    cpu
    ---
      Instance ID: cpu [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/cpu.d/conf.yaml.default
      Total Runs: 11
      Metric Samples: Last Run: 6, Total: 60
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-09-30 03:44:21.000000 UTC
      Last Successful Execution Date : 2020-09-30 03:44:21.000000 UTC


    disk (2.9.0)
    ------------
      Instance ID: disk:e5dffb8bef24336f [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/disk.d/conf.yaml.default
      Total Runs: 11
      Metric Samples: Last Run: 200, Total: 2,200
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 94ms
      Last Execution Date : 2020-09-30 03:44:13.000000 UTC
      Last Successful Execution Date : 2020-09-30 03:44:13.000000 UTC


    docker
    ------
      Instance ID: docker [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/docker.d/conf.yaml.default
      Total Runs: 11
      Metric Samples: Last Run: 118, Total: 1,298
      Events: Last Run: 0, Total: 5
      Service Checks: Last Run: 1, Total: 11
      Average Execution Time : 24ms
      Last Execution Date : 2020-09-30 03:44:20.000000 UTC
      Last Successful Execution Date : 2020-09-30 03:44:20.000000 UTC


    envoy (1.15.2)
    --------------
      Instance ID: envoy:e377cea6349bcca9 [ERROR]
      Configuration Source: file:/etc/datadog-agent/conf.d/envoy.d/conf.yaml
      Total Runs: 11
      Metric Samples: Last Run: 156, Total: 1,716
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 15ms
      Last Execution Date : 2020-09-30 03:44:07.000000 UTC
      Last Successful Execution Date : Never
      metadata:
        version.major: 1
        version.minor: 14
        version.patch: 3
        version.raw: 1.14.3-dev
        version.release: dev
        version.scheme: semver
      Error: '|_tags_|'
      Traceback (most recent call last):
        File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/base.py", line 822, in run
          self.check(instance)
        File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/envoy/envoy.py", line 93, in check
          metric, tags, method = parse_metric(envoy_metric)
        File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/envoy/parser.py", line 44, in parse_metric
          for tags in metric_mapping['|_tags_|']:
      KeyError: '|_tags_|'

    file_handle
    -----------
      Instance ID: file_handle [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/file_handle.d/conf.yaml.default
      Total Runs: 11
      Metric Samples: Last Run: 5, Total: 55
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-09-30 03:44:12.000000 UTC
      Last Successful Execution Date : 2020-09-30 03:44:12.000000 UTC


    io
    --
      Instance ID: io [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/io.d/conf.yaml.default
      Total Runs: 11
      Metric Samples: Last Run: 130, Total: 1,340
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-09-30 03:44:19.000000 UTC
      Last Successful Execution Date : 2020-09-30 03:44:19.000000 UTC


    istio (3.2.1)
    -------------
      Instance ID: istio:acc9a449e7fd2a02 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/istio.d/auto_conf.yaml
      Total Runs: 11
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 11
      Average Execution Time : 16ms
      Last Execution Date : 2020-09-30 03:44:14.000000 UTC
      Last Successful Execution Date : 2020-09-30 03:44:14.000000 UTC
      metadata:
        version.major: 1
        version.minor: 6
        version.patch: 4
        version.raw: 1.6.4
        version.scheme: semver


    load
    ----
      Instance ID: load [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/load.d/conf.yaml.default
      Total Runs: 11
      Metric Samples: Last Run: 6, Total: 66
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-09-30 03:44:11.000000 UTC
      Last Successful Execution Date : 2020-09-30 03:44:11.000000 UTC


    memory
    ------
      Instance ID: memory [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/memory.d/conf.yaml.default
      Total Runs: 11
      Metric Samples: Last Run: 17, Total: 187
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-09-30 03:44:18.000000 UTC
      Last Successful Execution Date : 2020-09-30 03:44:18.000000 UTC


    network (1.16.0)
    ----------------
      Instance ID: network:5c571333f400457d [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/network.d/conf.yaml.default
      Total Runs: 11
      Metric Samples: Last Run: 31, Total: 341
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 2ms
      Last Execution Date : 2020-09-30 03:44:10.000000 UTC
      Last Successful Execution Date : 2020-09-30 03:44:10.000000 UTC


    ntp
    ---
      Instance ID: ntp:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/ntp.d/conf.yaml.default
      Total Runs: 1
      Metric Samples: Last Run: 1, Total: 1
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 1
      Average Execution Time : 287ms
      Last Execution Date : 2020-09-30 03:41:37.000000 UTC
      Last Successful Execution Date : 2020-09-30 03:41:37.000000 UTC


    uptime
    ------
      Instance ID: uptime [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/uptime.d/conf.yaml.default
      Total Runs: 11
      Metric Samples: Last Run: 1, Total: 11
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-09-30 03:44:17.000000 UTC
      Last Successful Execution Date : 2020-09-30 03:44:17.000000 UTC

========
JMXFetch
========

  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    CheckRunsV1: 11
    Connections: 0
    Containers: 0
    Dropped: 0
    DroppedOnInput: 0
    Events: 0
    HostMetadata: 0
    IntakeV1: 4
    Metadata: 0
    Pods: 0
    Processes: 0
    RTContainers: 0
    RTProcesses: 0
    Requeued: 0
    Retried: 0
    RetryQueueSize: 0
    Series: 0
    ServiceChecks: 0
    SketchSeries: 0
    Success: 26
    TimeseriesV1: 11

  API Keys status
  ===============
    API key ending with <redacted>: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.com - API Key ending with:
      - <redacted>

==========
Logs Agent
==========


  Logs Agent is not running

=========
APM Agent
=========
  Status: Running
  Pid: 348
  Uptime: 167 seconds
  Mem alloc: 8,072,824 bytes
  Hostname: <redacted>
  Receiver: 0.0.0.0:8126
  Endpoints:
    https://trace.agent.datadoghq.com

  Receiver (previous minute)
  ==========================
    No traces received in the previous minute.
    Default priority sampling rate: 100.0%

  Writer (previous minute)
  ========================
    Traces: 0 payloads, 0 traces, 0 events, 0 bytes
    Stats: 0 payloads, 0 stats buckets, 0 bytes

=========
Aggregator
=========
  Checks Metric Sample: 7,518
  Dogstatsd Metric Sample: 1,177
  Event: 6
  Events Flushed: 6
  Number Of Flushes: 11
  Series Flushed: 5,717
  Service Check: 145
  Service Checks Flushed: 152

=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 1,176
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 86,482
  Udp Packet Reading Errors: 0
  Udp Packets: 122
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 0

Additional environment details (Operating System, Cloud provider, etc):

  • Datadog agent and Istio running in Docker containers, launched by docker-compose. Agent is configured to access Istio Envoy metrics from admin /stats endpoint.
  • I'm running Istio with Docker image and tag: istio/proxyv2:1.6.4, datadog-agent of datadog/agent:6.20.0.

Envoy Agent Config

init_config:

instances:
  ## @param stats_url - string - required
  ## The admin endpoint to connect to. It must be accessible:
  ## https://www.envoyproxy.io/docs/envoy/latest/operations/admin
  ## Add a `?usedonly` on the end if you wish to ignore
  ## unused metrics instead of reporting them as `0`.
  #
  - stats_url: http://localhost:15000/stats

Istio Agent Config

init_config:

## Every instance is scheduled independent of the others.
#
instances:

    ## @param send_histograms_buckets - boolean - required
    ## Set send_histograms_buckets to true to send the histograms bucket from Istio.
    #
  - # send_histograms_buckets: true

    ## @param send_distribution_buckets - boolean - optional
    ## Set send_distribution_buckets to true to send and convert OpenMetrics histograms to Distribution metrics.
    #
    # send_distribution_buckets: true

    ## @param istio_mesh_endpoint - string - optional
    ## To enable Istio metrics you must specify the url exposing the API.
    #
    istiod_endpoint: http://localhost:15090/stats/prometheus

Additional information you deem important (e.g. issue happens only occasionally):
I read through and ran functions in envoy check parser.py, and noticed when these metrics are passed into this function, parse_metric function crashes. I tried adding these metric names under Envoy agent config metric_blacklist key, and confirmed the agent can launch successfully without crashing.

server.main_thread.watchdog_mega_miss
server.main_thread.watchdog_miss
server.worker_0.watchdog_mega_miss
server.worker_0.watchdog_miss
server.worker_1.watchdog_mega_miss
server.worker_1.watchdog_miss
server.worker_10.watchdog_mega_miss
server.worker_10.watchdog_miss
server.worker_11.watchdog_mega_miss
server.worker_11.watchdog_miss
server.worker_12.watchdog_mega_miss
server.worker_12.watchdog_miss
server.worker_13.watchdog_mega_miss
server.worker_13.watchdog_miss
server.worker_14.watchdog_mega_miss
server.worker_14.watchdog_miss
server.worker_15.watchdog_mega_miss
server.worker_15.watchdog_miss
server.worker_16.watchdog_mega_miss
server.worker_16.watchdog_miss
server.worker_17.watchdog_mega_miss
server.worker_17.watchdog_miss
server.worker_18.watchdog_mega_miss
server.worker_18.watchdog_miss
server.worker_19.watchdog_mega_miss
server.worker_19.watchdog_miss
server.worker_2.watchdog_mega_miss
server.worker_2.watchdog_miss
server.worker_20.watchdog_mega_miss
server.worker_20.watchdog_miss
server.worker_21.watchdog_mega_miss
server.worker_21.watchdog_miss
server.worker_22.watchdog_mega_miss
server.worker_22.watchdog_miss
server.worker_23.watchdog_mega_miss
server.worker_23.watchdog_miss
server.worker_24.watchdog_mega_miss
server.worker_24.watchdog_miss
server.worker_25.watchdog_mega_miss
server.worker_25.watchdog_miss
server.worker_26.watchdog_mega_miss
server.worker_26.watchdog_miss
server.worker_27.watchdog_mega_miss
server.worker_27.watchdog_miss
server.worker_28.watchdog_mega_miss
server.worker_28.watchdog_miss
server.worker_29.watchdog_mega_miss
server.worker_29.watchdog_miss
server.worker_3.watchdog_mega_miss
server.worker_3.watchdog_miss
server.worker_30.watchdog_mega_miss
server.worker_30.watchdog_miss
server.worker_31.watchdog_mega_miss
server.worker_31.watchdog_miss
server.worker_4.watchdog_mega_miss
server.worker_4.watchdog_miss
server.worker_5.watchdog_mega_miss
server.worker_5.watchdog_miss
server.worker_6.watchdog_mega_miss
server.worker_6.watchdog_miss
server.worker_7.watchdog_mega_miss
server.worker_7.watchdog_miss
server.worker_8.watchdog_mega_miss
server.worker_8.watchdog_miss
server.worker_9.watchdog_mega_miss
server.worker_9.watchdog_miss
wasm.envoy.wasm.runtime.null.active
wasm.envoy.wasm.runtime.null.created
wasm_vm.envoy.wasm.runtime.null.active
wasm_vm.envoy.wasm.runtime.null.cloned
wasm_vm.envoy.wasm.runtime.null.created
@max0ne
Copy link
Author

max0ne commented Sep 30, 2020

A complete list of Envoy metrics:

$ curl http://localhost:15000/stats
cluster.xds-grpc.assignment_stale: 0
cluster.xds-grpc.assignment_timeout_received: 0
cluster.xds-grpc.bind_errors: 0
cluster.xds-grpc.circuit_breakers.default.cx_open: 0
cluster.xds-grpc.circuit_breakers.default.cx_pool_open: 0
cluster.xds-grpc.circuit_breakers.default.rq_open: 0
cluster.xds-grpc.circuit_breakers.default.rq_pending_open: 0
cluster.xds-grpc.circuit_breakers.default.rq_retry_open: 0
cluster.xds-grpc.circuit_breakers.high.cx_open: 0
cluster.xds-grpc.circuit_breakers.high.cx_pool_open: 0
cluster.xds-grpc.circuit_breakers.high.rq_open: 0
cluster.xds-grpc.circuit_breakers.high.rq_pending_open: 0
cluster.xds-grpc.circuit_breakers.high.rq_retry_open: 0
cluster.xds-grpc.client_ssl_socket_factory.downstream_context_secrets_not_ready: 0
cluster.xds-grpc.client_ssl_socket_factory.ssl_context_update_by_sds: 1
cluster.xds-grpc.client_ssl_socket_factory.upstream_context_secrets_not_ready: 0
cluster.xds-grpc.default.total_match_count: 211
cluster.xds-grpc.http2.dropped_headers_with_underscores: 0
cluster.xds-grpc.http2.header_overflow: 0
cluster.xds-grpc.http2.headers_cb_no_stream: 0
cluster.xds-grpc.http2.inbound_empty_frames_flood: 0
cluster.xds-grpc.http2.inbound_priority_frames_flood: 0
cluster.xds-grpc.http2.inbound_window_update_frames_flood: 0
cluster.xds-grpc.http2.outbound_control_flood: 0
cluster.xds-grpc.http2.outbound_flood: 0
cluster.xds-grpc.http2.pending_send_bytes: 0
cluster.xds-grpc.http2.requests_rejected_with_underscores_in_headers: 0
cluster.xds-grpc.http2.rx_messaging_error: 0
cluster.xds-grpc.http2.rx_reset: 0
cluster.xds-grpc.http2.streams_active: 1
cluster.xds-grpc.http2.too_many_header_frames: 0
cluster.xds-grpc.http2.trailers: 0
cluster.xds-grpc.http2.tx_flush_timeout: 0
cluster.xds-grpc.http2.tx_reset: 0
cluster.xds-grpc.internal.upstream_rq_200: 1
cluster.xds-grpc.internal.upstream_rq_2xx: 1
cluster.xds-grpc.internal.upstream_rq_503: 2
cluster.xds-grpc.internal.upstream_rq_5xx: 2
cluster.xds-grpc.internal.upstream_rq_completed: 3
cluster.xds-grpc.lb_healthy_panic: 2
cluster.xds-grpc.lb_local_cluster_not_ok: 0
cluster.xds-grpc.lb_recalculate_zone_structures: 0
cluster.xds-grpc.lb_subsets_active: 0
cluster.xds-grpc.lb_subsets_created: 0
cluster.xds-grpc.lb_subsets_fallback: 0
cluster.xds-grpc.lb_subsets_fallback_panic: 0
cluster.xds-grpc.lb_subsets_removed: 0
cluster.xds-grpc.lb_subsets_selected: 0
cluster.xds-grpc.lb_zone_cluster_too_small: 0
cluster.xds-grpc.lb_zone_no_capacity_left: 0
cluster.xds-grpc.lb_zone_number_differs: 0
cluster.xds-grpc.lb_zone_routing_all_directly: 0
cluster.xds-grpc.lb_zone_routing_cross_zone: 0
cluster.xds-grpc.lb_zone_routing_sampled: 0
cluster.xds-grpc.max_host_weight: 1
cluster.xds-grpc.membership_change: 1
cluster.xds-grpc.membership_degraded: 0
cluster.xds-grpc.membership_excluded: 0
cluster.xds-grpc.membership_healthy: 1
cluster.xds-grpc.membership_total: 1
cluster.xds-grpc.original_dst_host_invalid: 0
cluster.xds-grpc.retry_or_shadow_abandoned: 0
cluster.xds-grpc.ssl.ciphers.ECDHE-RSA-AES128-GCM-SHA256: 1
cluster.xds-grpc.ssl.connection_error: 0
cluster.xds-grpc.ssl.curves.X25519: 1
cluster.xds-grpc.ssl.fail_verify_cert_hash: 0
cluster.xds-grpc.ssl.fail_verify_error: 0
cluster.xds-grpc.ssl.fail_verify_no_cert: 0
cluster.xds-grpc.ssl.fail_verify_san: 0
cluster.xds-grpc.ssl.handshake: 1
cluster.xds-grpc.ssl.no_certificate: 0
cluster.xds-grpc.ssl.session_reused: 0
cluster.xds-grpc.ssl.sigalgs.rsa_pss_rsae_sha256: 1
cluster.xds-grpc.ssl.versions.TLSv1.2: 1
cluster.xds-grpc.update_attempt: 211
cluster.xds-grpc.update_empty: 0
cluster.xds-grpc.update_failure: 0
cluster.xds-grpc.update_no_rebuild: 210
cluster.xds-grpc.update_success: 211
cluster.xds-grpc.upstream_cx_active: 1
cluster.xds-grpc.upstream_cx_close_notify: 0
cluster.xds-grpc.upstream_cx_connect_attempts_exceeded: 0
cluster.xds-grpc.upstream_cx_connect_fail: 0
cluster.xds-grpc.upstream_cx_connect_timeout: 0
cluster.xds-grpc.upstream_cx_destroy: 0
cluster.xds-grpc.upstream_cx_destroy_local: 0
cluster.xds-grpc.upstream_cx_destroy_local_with_active_rq: 0
cluster.xds-grpc.upstream_cx_destroy_remote: 0
cluster.xds-grpc.upstream_cx_destroy_remote_with_active_rq: 0
cluster.xds-grpc.upstream_cx_destroy_with_active_rq: 0
cluster.xds-grpc.upstream_cx_http1_total: 0
cluster.xds-grpc.upstream_cx_http2_total: 1
cluster.xds-grpc.upstream_cx_idle_timeout: 0
cluster.xds-grpc.upstream_cx_max_requests: 1
cluster.xds-grpc.upstream_cx_none_healthy: 2
cluster.xds-grpc.upstream_cx_overflow: 0
cluster.xds-grpc.upstream_cx_pool_overflow: 0
cluster.xds-grpc.upstream_cx_protocol_error: 0
cluster.xds-grpc.upstream_cx_rx_bytes_buffered: 43
cluster.xds-grpc.upstream_cx_rx_bytes_total: 22643081
cluster.xds-grpc.upstream_cx_total: 1
cluster.xds-grpc.upstream_cx_tx_bytes_buffered: 0
cluster.xds-grpc.upstream_cx_tx_bytes_total: 7580050
cluster.xds-grpc.upstream_flow_control_backed_up_total: 0
cluster.xds-grpc.upstream_flow_control_drained_total: 0
cluster.xds-grpc.upstream_flow_control_paused_reading_total: 0
cluster.xds-grpc.upstream_flow_control_resumed_reading_total: 0
cluster.xds-grpc.upstream_internal_redirect_failed_total: 0
cluster.xds-grpc.upstream_internal_redirect_succeeded_total: 0
cluster.xds-grpc.upstream_rq_200: 1
cluster.xds-grpc.upstream_rq_2xx: 1
cluster.xds-grpc.upstream_rq_503: 2
cluster.xds-grpc.upstream_rq_5xx: 2
cluster.xds-grpc.upstream_rq_active: 1
cluster.xds-grpc.upstream_rq_cancelled: 0
cluster.xds-grpc.upstream_rq_completed: 3
cluster.xds-grpc.upstream_rq_maintenance_mode: 0
cluster.xds-grpc.upstream_rq_pending_active: 0
cluster.xds-grpc.upstream_rq_pending_failure_eject: 0
cluster.xds-grpc.upstream_rq_pending_overflow: 0
cluster.xds-grpc.upstream_rq_pending_total: 1
cluster.xds-grpc.upstream_rq_per_try_timeout: 0
cluster.xds-grpc.upstream_rq_retry: 0
cluster.xds-grpc.upstream_rq_retry_limit_exceeded: 0
cluster.xds-grpc.upstream_rq_retry_overflow: 0
cluster.xds-grpc.upstream_rq_retry_success: 0
cluster.xds-grpc.upstream_rq_rx_reset: 0
cluster.xds-grpc.upstream_rq_timeout: 0
cluster.xds-grpc.upstream_rq_total: 1
cluster.xds-grpc.upstream_rq_tx_reset: 0
cluster.xds-grpc.version: 0
cluster_manager.active_clusters: 278
cluster_manager.cds.init_fetch_timeout: 0
cluster_manager.cds.update_attempt: 72
cluster_manager.cds.update_failure: 0
cluster_manager.cds.update_rejected: 0
cluster_manager.cds.update_success: 71
cluster_manager.cds.update_time: 1601438335821
cluster_manager.cds.version: 1420590272502474275
cluster_manager.cluster_added: 278
cluster_manager.cluster_modified: 49
cluster_manager.cluster_removed: 0
cluster_manager.cluster_updated: 585
cluster_manager.cluster_updated_via_merge: 0
cluster_manager.update_merge_cancelled: 0
cluster_manager.update_out_of_merge_window: 0
cluster_manager.warming_clusters: 0
component.proxy.tag.1.6.4;._istio_build: 1
listener_manager.lds.init_fetch_timeout: 0
listener_manager.lds.update_attempt: 72
listener_manager.lds.update_failure: 0
listener_manager.lds.update_rejected: 0
listener_manager.lds.update_success: 71
listener_manager.lds.update_time: 1601438335900
listener_manager.lds.version: 1420590272502474275
listener_manager.listener_added: 5
listener_manager.listener_create_failure: 0
listener_manager.listener_create_success: 160
listener_manager.listener_in_place_updated: 0
listener_manager.listener_modified: 0
listener_manager.listener_removed: 0
listener_manager.listener_stopped: 0
listener_manager.total_filter_chains_draining: 0
listener_manager.total_listeners_active: 5
listener_manager.total_listeners_draining: 0
listener_manager.total_listeners_warming: 0
listener_manager.workers_started: 1
server.concurrency: 32
server.days_until_first_cert_expiring: 0
server.debug_assertion_failures: 0
server.dynamic_unknown_fields: 0
server.hot_restart_epoch: 0
server.hot_restart_generation: 1
server.live: 1
server.main_thread.watchdog_mega_miss: 0
server.main_thread.watchdog_miss: 1
server.memory_allocated: 51258536
server.memory_heap_size: 68157440
server.memory_physical_size: 73269248
server.parent_connections: 0
server.state: 0
server.static_unknown_fields: 0
server.stats_recent_lookups: 0
server.total_connections: 0
server.uptime: 1050
server.version: 8432902
server.watchdog_mega_miss: 0
server.watchdog_miss: 1
server.worker_0.watchdog_mega_miss: 0
server.worker_0.watchdog_miss: 0
server.worker_1.watchdog_mega_miss: 0
server.worker_1.watchdog_miss: 0
server.worker_10.watchdog_mega_miss: 0
server.worker_10.watchdog_miss: 0
server.worker_11.watchdog_mega_miss: 0
server.worker_11.watchdog_miss: 0
server.worker_12.watchdog_mega_miss: 0
server.worker_12.watchdog_miss: 0
server.worker_13.watchdog_mega_miss: 0
server.worker_13.watchdog_miss: 0
server.worker_14.watchdog_mega_miss: 0
server.worker_14.watchdog_miss: 0
server.worker_15.watchdog_mega_miss: 0
server.worker_15.watchdog_miss: 0
server.worker_16.watchdog_mega_miss: 0
server.worker_16.watchdog_miss: 0
server.worker_17.watchdog_mega_miss: 0
server.worker_17.watchdog_miss: 0
server.worker_18.watchdog_mega_miss: 0
server.worker_18.watchdog_miss: 0
server.worker_19.watchdog_mega_miss: 0
server.worker_19.watchdog_miss: 0
server.worker_2.watchdog_mega_miss: 0
server.worker_2.watchdog_miss: 0
server.worker_20.watchdog_mega_miss: 0
server.worker_20.watchdog_miss: 0
server.worker_21.watchdog_mega_miss: 0
server.worker_21.watchdog_miss: 0
server.worker_22.watchdog_mega_miss: 0
server.worker_22.watchdog_miss: 0
server.worker_23.watchdog_mega_miss: 0
server.worker_23.watchdog_miss: 0
server.worker_24.watchdog_mega_miss: 0
server.worker_24.watchdog_miss: 0
server.worker_25.watchdog_mega_miss: 0
server.worker_25.watchdog_miss: 0
server.worker_26.watchdog_mega_miss: 0
server.worker_26.watchdog_miss: 0
server.worker_27.watchdog_mega_miss: 0
server.worker_27.watchdog_miss: 0
server.worker_28.watchdog_mega_miss: 0
server.worker_28.watchdog_miss: 0
server.worker_29.watchdog_mega_miss: 0
server.worker_29.watchdog_miss: 0
server.worker_3.watchdog_mega_miss: 0
server.worker_3.watchdog_miss: 0
server.worker_30.watchdog_mega_miss: 0
server.worker_30.watchdog_miss: 0
server.worker_31.watchdog_mega_miss: 0
server.worker_31.watchdog_miss: 0
server.worker_4.watchdog_mega_miss: 0
server.worker_4.watchdog_miss: 0
server.worker_5.watchdog_mega_miss: 0
server.worker_5.watchdog_miss: 0
server.worker_6.watchdog_mega_miss: 0
server.worker_6.watchdog_miss: 0
server.worker_7.watchdog_mega_miss: 0
server.worker_7.watchdog_miss: 0
server.worker_8.watchdog_mega_miss: 0
server.worker_8.watchdog_miss: 0
server.worker_9.watchdog_mega_miss: 0
server.worker_9.watchdog_miss: 0
wasm.envoy.wasm.runtime.null.active: 102
wasm.envoy.wasm.runtime.null.created: 102
wasm_filter.stats_filter.cache.hit.metric_cache_count: 0
wasm_filter.stats_filter.cache.miss.metric_cache_count: 0
wasm_vm.envoy.wasm.runtime.null.active: 102
wasm_vm.envoy.wasm.runtime.null.cloned: 0
wasm_vm.envoy.wasm.runtime.null.created: 102
cluster.xds-grpc.upstream_cx_connect_ms: P0(nan,10.0) P25(nan,10.25) P50(nan,10.5) P75(nan,10.75) P90(nan,10.9) P95(nan,10.95) P99(nan,10.99) P99.5(nan,10.995) P99.9(nan,10.999) P100(nan,11.0)
cluster.xds-grpc.upstream_cx_length_ms: No recorded values
server.initialization_time_ms: P0(nan,1500.0) P25(nan,1525.0) P50(nan,1550.0) P75(nan,1575.0) P90(nan,1590.0) P95(nan,1595.0) P99(nan,1599.0) P99.5(nan,1599.5) P99.9(nan,1599.9) P100(nan,1600.0)

@ofek ofek mentioned this issue Oct 5, 2020
Merged
@ofek ofek closed this as completed in #7717 Oct 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ofek ofek

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Properly handle a parsing edge case DataDog/integrations-core
2 participants
@max0ne @ofek