[Monitoring] Ensure Elasticsearch parts of Stack Monitoring work with metricbeat-* indices

[chrisronline]

Relates to #73864
Relates to elastic/beats#24427

This PR is meant to update all existing Stack Monitoring UI code to correctly work off metricbeat-* and .monitoring-* indices for Elasticsearch-related areas of code. It is an extension of #88212 as it identifies the fields missed by actually using a working Metricbeat module (previously, we had to mock the output of the module which was error-prone).

This effort is hard to guarantee 100% completion because we need to ensure every part of the front-end works properly with both data sets. We have two main ways to test this:

1. Spot-check manually that all charts/data values are properly presented in the UI
2. Leverage our existing functional test suites

The main issue with 2 is that our existing functional test suites use archive data which is in .monitoring-* indices so they aren't very helpful in ensuring the metricbeat-* data path works (but are helpful in determining the .monitoring-* data path still does work) so I did some work to build archive data using metricbeat-* indices.

This work was done in several parts:

Part 1: Write a script that identifies all known aliases that currently exist in the metricbeat-* indices and write it to a file: https://gist.github.com/chrisronline/d24ec862efdb0cfa331c7ef785241116

Part 2: Modify the rebuild all function of the es_archiver script to build out metricbeat-* indices from the existing .monitoring-* data archives, using the known aliases as well as some custom logic: https://github.com/elastic/kibana/pull/90838/files#diff-b0251a00b48f25e70b98d6741ae88948b22f9220a5ba5914aaaab465038da659 (This script is ran like: node ../scripts/es_archiver.js rebuild-all)

Part 3: Duplicate existing test suites and use the newly created archives based off metricbeat-* indices: https://github.com/elastic/kibana/pull/90838/files#diff-36d8e173c4e5a530dd9d5d68faca3cfd3f69f3803f80efda74c1f8bc562f033b

Note: elastic/elasticsearch#71233 needs to be merged before all functional test will consistently pass

[chrisronline]

Closing in favor of a feature branch in the elastic repo: #96205

[kibanamachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: c1d505d
• Storybooks Preview
• Pipeline Steps (look for red circles / failed steps)
• Interpreting CI Failures
• Documentation Changes

Failed CI Steps

• Jest Unit Tests
• Execute xpack-kibana-ciGroup1
• Execute xpack-kibana-ciGroup1
• Execute xpack-kibana-ciGroup6

---

Test Failures

Kibana Pipeline / jest / Jest Tests.x-pack/plugins/monitoring/server/lib/metrics.Metrics should export metric objects that match a snapshot

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

Error: expect(received).toMatchSnapshot()

Snapshot name: `Metrics should export metric objects that match a snapshot 1`

- Snapshot  - 0
+ Received  + 1

@@ -3350,10 +3350,11 @@
            "unit": "1s",
          },
        },
      },
      "app": "logstash",
+     "debug": true,
      "derivative": true,
      "description": "Number of events received per second by all Logstash nodes at the inputs stage.",
      "field": "logstash_stats.events.in",
      "format": "0,0.[00]",
      "label": "Events Received Rate",
    at Object.<anonymous> (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/x-pack/plugins/monitoring/server/lib/metrics/metrics.test.js:12:21)
    at Promise.then.completed (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/utils.js:276:28)
    at new Promise (<anonymous>)
    at callAsyncCircusFn (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/utils.js:216:10)
    at _callCircusTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:212:40)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at _runTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:149:3)
    at _runTestsForDescribeBlock (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:63:9)
    at _runTestsForDescribeBlock (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:57:9)
    at run (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:25:3)
    at runAndTransformResultsToJestFormat (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapterInit.js:176:21)
    at jestAdapter (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapter.js:109:19)
    at runTestInternal (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/runTest.js:380:16)
    at runTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/runTest.js:472:34)
    at Object.worker (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/testWorker.js:133:12)

---

Kibana Pipeline / general / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/monitoring/elasticsearch/index_detail_mb·js.Monitoring app Elasticsearch index detail mb Active Indices "before all" hook for "should have an index summary with green status index with full shard allocation"

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: Monitoring app
[00:00:00]           └-> "before all" hook in "Monitoring app"
[00:13:55]           └-: Elasticsearch index detail mb
[00:13:55]             └-> "before all" hook in "Elasticsearch index detail mb"
[00:13:55]             └-: Active Indices
[00:13:55]               └-> "before all" hook for "should have an index summary with green status index with full shard allocation"
[00:13:55]               └-> "before all" hook for "should have an index summary with green status index with full shard allocation"
[00:13:55]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Loading "mappings.json"
[00:13:55]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Loading "data.json.gz"
[00:13:55]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0] creating index, cause [api], templates [], shards [1]/[1]
[00:13:56]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index "metricbeat-8.0.0"
[00:13:56]                 │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] "metricbeat-8.0.0" settings {"index":{"codec":"best_compression","lifecycle":{"name":"metricbeat","rollover_alias":"metricbeat-8.0.0"},"mapping":{"total_fields":{"limit":"10000"}},"max_docvalue_fields_search":"200","number_of_replicas":"1","number_of_shards":"1","query":{"default_field":["message","tags","agent.ephemeral_id","agent.id","agent.name","agent.type","agent.version","as.organization.name","client.address","client.as.organization.name","client.domain","client.geo.city_name","client.geo.continent_name","client.geo.country_iso_code","client.geo.country_name","client.geo.name","client.geo.region_iso_code","client.geo.region_name","client.mac","client.registered_domain","client.top_level_domain","client.user.domain","client.user.email","client.user.full_name","client.user.group.domain","client.user.group.id","client.user.group.name","client.user.hash","client.user.id","client.user.name","cloud.account.id","cloud.availability_zone","cloud.instance.id","cloud.instance.name","cloud.machine.type","cloud.provider","cloud.region","container.id","container.image.name","container.image.tag","container.name","container.runtime","destination.address","destination.as.organization.name","destination.domain","destination.geo.city_name","destination.geo.continent_name","destination.geo.country_iso_code","destination.geo.country_name","destination.geo.name","destination.geo.region_iso_code","destination.geo.region_name","destination.mac","destination.registered_domain","destination.top_level_domain","destination.user.domain","destination.user.email","destination.user.full_name","destination.user.group.domain","destination.user.group.id","destination.user.group.name","destination.user.hash","destination.user.id","destination.user.name","dns.answers.class","dns.answers.data","dns.answers.name","dns.answers.type","dns.header_flags","dns.id","dns.op_code","dns.question.class","dns.question.name","dns.question.registered_domain","dns.question.subdomain","dns.question.top_level_domain","dns.question.type","dns.response_code","dns.type","ecs.version","error.code","error.id","error.message","error.stack_trace","error.type","event.action","event.category","event.code","event.dataset","event.hash","event.id","event.kind","event.module","event.original","event.outcome","event.provider","event.timezone","event.type","file.device","file.directory","file.extension","file.gid","file.group","file.hash.md5","file.hash.sha1","file.hash.sha256","file.hash.sha512","file.inode","file.mode","file.name","file.owner","file.path","file.target_path","file.type","file.uid","geo.city_name","geo.continent_name","geo.country_iso_code","geo.country_name","geo.name","geo.region_iso_code","geo.region_name","group.domain","group.id","group.name","hash.md5","hash.sha1","hash.sha256","hash.sha512","host.architecture","host.geo.city_name","host.geo.continent_name","host.geo.country_iso_code","host.geo.country_name","host.geo.name","host.geo.region_iso_code","host.geo.region_name","host.hostname","host.id","host.mac","host.name","host.os.family","host.os.full","host.os.kernel","host.os.name","host.os.platform","host.os.version","host.type","host.user.domain","host.user.email","host.user.full_name","host.user.group.domain","host.user.group.id","host.user.group.name","host.user.hash","host.user.id","host.user.name","http.request.body.content","http.request.method","http.request.referrer","http.response.body.content","http.version","log.level","log.logger","log.origin.file.name","log.origin.function","log.original","log.syslog.facility.name","log.syslog.severity.name","network.application","network.community_id","network.direction","network.iana_number","network.name","network.protocol","network.transport","network.type","observer.geo.city_name","observer.geo.continent_name","observer.geo.country_iso_code","observer.geo.country_name","observer.geo.name","observer.geo.region_iso_code","observer.geo.region_name","observer.hostname","observer.mac","observer.name","observer.os.family","observer.os.full","observer.os.kernel","observer.os.name","observer.os.platform","observer.os.version","observer.product","observer.serial_number","observer.type","observer.vendor","observer.version","organization.id","organization.name","os.family","os.full","os.kernel","os.name","os.platform","os.version","package.architecture","package.checksum","package.description","package.install_scope","package.license","package.name","package.path","package.version","process.args","text","process.executable","process.hash.md5","process.hash.sha1","process.hash.sha256","process.hash.sha512","process.name","text","text","text","text","text","process.thread.name","process.title","process.working_directory","server.address","server.as.organization.name","server.domain","server.geo.city_name","server.geo.continent_name","server.geo.country_iso_code","server.geo.country_name","server.geo.name","server.geo.region_iso_code","server.geo.region_name","server.mac","server.registered_domain","server.top_level_domain","server.user.domain","server.user.email","server.user.full_name","server.user.group.domain","server.user.group.id","server.user.group.name","server.user.hash","server.user.id","server.user.name","service.ephemeral_id","service.id","service.name","service.node.name","service.state","service.type","service.version","source.address","source.as.organization.name","source.domain","source.geo.city_name","source.geo.continent_name","source.geo.country_iso_code","source.geo.country_name","source.geo.name","source.geo.region_iso_code","source.geo.region_name","source.mac","source.registered_domain","source.top_level_domain","source.user.domain","source.user.email","source.user.full_name","source.user.group.domain","source.user.group.id","source.user.group.name","source.user.hash","source.user.id","source.user.name","threat.framework","threat.tactic.id","threat.tactic.name","threat.tactic.reference","threat.technique.id","threat.technique.name","threat.technique.reference","tracing.trace.id","tracing.transaction.id","url.domain","url.extension","url.fragment","url.full","url.original","url.password","url.path","url.query","url.registered_domain","url.scheme","url.top_level_domain","url.username","user.domain","user.email","user.full_name","user.group.domain","user.group.id","user.group.name","user.hash","user.id","user.name","user_agent.device.name","user_agent.name","text","user_agent.original","user_agent.os.family","user_agent.os.full","user_agent.os.kernel","user_agent.os.name","user_agent.os.platform","user_agent.os.version","user_agent.version","text","agent.hostname","timeseries.instance","cloud.project.id","cloud.image.id","host.os.build","host.os.codename","kubernetes.pod.name","kubernetes.pod.uid","kubernetes.namespace","kubernetes.node.name","kubernetes.replicaset.name","kubernetes.deployment.name","kubernetes.statefulset.name","kubernetes.container.name","kubernetes.container.image","jolokia.agent.version","jolokia.agent.id","jolokia.server.product","jolokia.server.version","jolokia.server.vendor","jolokia.url","metricset.name","service.address","service.hostname","type","systemd.fragment_path","systemd.unit","aerospike.namespace.name","aerospike.namespace.node.host","aerospike.namespace.node.name","apache.status.hostname","beat.id","beat.type","beat.state.service.id","beat.state.service.name","beat.state.service.version","beat.state.input.names","beat.state.beat.host","beat.state.beat.name","beat.state.beat.type","beat.state.beat.uuid","beat.state.beat.version","beat.state.cluster.uuid","beat.state.host.containerized","beat.state.host.os.kernel","beat.state.host.os.name","beat.state.host.os.platform","beat.state.host.os.version","beat.state.module.names","beat.state.output.name","beat.state.queue.name","beat.stats.beat.name","beat.stats.beat.host","beat.stats.beat.type","beat.stats.beat.uuid","beat.stats.beat.version","beat.stats.info.ephemeral_id","beat.stats.libbeat.output.type","ceph.cluster_health.overall_status","ceph.cluster_health.timechecks.round.status","ceph.mgr_osd_pool_stats.pool_name","ceph.monitor_health.health","ceph.monitor_health.name","ceph.osd_df.name","ceph.osd_df.device_class","ceph.osd_tree.name","ceph.osd_tree.type","ceph.osd_tree.children","ceph.osd_tree.status","ceph.osd_tree.device_class","ceph.osd_tree.father","ceph.pool_disk.name","couchbase.bucket.name","couchbase.bucket.type","couchbase.node.hostname","docker.container.command","docker.container.status","docker.container.tags","docker.event.status","docker.event.id","docker.event.from","docker.event.type","docker.event.action","docker.event.actor.id","docker.healthcheck.status","docker.healthcheck.event.output","docker.image.id.current","docker.image.id.parent","docker.image.tags","docker.info.id","docker.network.interface","elasticsearch.cluster.name","elasticsearch.cluster.id","elasticsearch.cluster.state.id","elasticsearch.node.id","elasticsearch.node.name","elasticsearch.ccr.remote_cluster","elasticsearch.ccr.leader.index","elasticsearch.ccr.follower.index","elasticsearch.cluster.stats.version","elasticsearch.cluster.stats.state.nodes_hash","elasticsearch.cluster.stats.state.master_node","elasticsearch.cluster.stats.state.version","elasticsearch.cluster.stats.state.state_uuid","elasticsearch.cluster.stats.status","elasticsearch.cluster.stats.license.status","elasticsearch.cluster.stats.license.type","elasticsearch.enrich.executing_policy.name","elasticsearch.enrich.executing_policy.task.task","elasticsearch.enrich.executing_policy.task.action","elasticsearch.enrich.executing_policy.task.parent_task_id","elasticsearch.index.uuid","elasticsearch.index.status","elasticsearch.index.name","elasticsearch.index.recovery.index.files.percent","elasticsearch.index.recovery.name","elasticsearch.index.recovery.type","elasticsearch.index.recovery.stage","elasticsearch.index.recovery.translog.percent","elasticsearch.index.recovery.target.transport_address","elasticsearch.index.recovery.target.id","elasticsearch.index.recovery.target.host","elasticsearch.index.recovery.target.name","elasticsearch.index.recovery.source.transport_address","elasticsearch.index.recovery.source.id","elasticsearch.index.recovery.source.host","elasticsearch.index.recovery.source.name","elasticsearch.ml.job.id","elasticsearch.ml.job.state","elasticsearch.ml.job.model_size.memory_status","elasticsearch.node.version","elasticsearch.node.jvm.version","elasticsearch.node.stats.os.cgroup.memory.control_group","elasticsearch.cluster.pending_task.source","elasticsearch.shard.state","elasticsearch.shard.relocating_node.name","elasticsearch.shard.relocating_node.id","elasticsearch.shard.source_node.name","elasticsearch.shard.source_node.uuid","etcd.api_version","etcd.leader.leader","etcd.self.id","etcd.self.leaderinfo.leader","etcd.self.leaderinfo.starttime","etcd.self.leaderinfo.uptime","etcd.self.name","etcd.self.starttime","etcd.self.state","golang.expvar.cmdline","golang.heap.cmdline","graphite.server.example","haproxy.stat.status","haproxy.stat.service_name","haproxy.stat.cookie","haproxy.stat.load_balancing_algorithm","haproxy.stat.check.status","haproxy.stat.check.health.last","haproxy.stat.proxy.name","haproxy.stat.proxy.mode","haproxy.stat.agent.status","haproxy.stat.agent.description","haproxy.stat.agent.check.description","haproxy.stat.source.address","http.response.code","http.response.phrase","kafka.broker.address","kafka.topic.name","kafka.partition.topic_id","kafka.partition.topic_broker_id","kafka.broker.mbean","kafka.consumer.mbean","kafka.consumergroup.broker.address","kafka.consumergroup.id","kafka.consumergroup.topic","kafka.consumergroup.meta","kafka.consumergroup.client.id","kafka.consumergroup.client.host","kafka.consumergroup.client.member_id","kafka.partition.topic.name","kafka.partition.broker.address","kafka.producer.mbean","kibana.settings.uuid","kibana.settings.name","kibana.settings.index","kibana.settings.host","kibana.settings.transport_address","kibana.settings.version","kibana.settings.status","kibana.settings.locale","kibana.stats.kibana.status","kibana.stats.usage.index","kibana.stats.name","kibana.stats.index","kibana.stats.host.name","kibana.stats.status","kibana.stats.os.distro","kibana.stats.os.distroRelease","kibana.stats.os.platform","kibana.stats.os.platformRelease","kibana.status.name","kibana.status.status.overall.state","kubernetes.apiserver.request.client","kubernetes.apiserver.request.resource","kubernetes.apiserver.request.subresource","kubernetes.apiserver.request.scope","kubernetes.apiserver.request.verb","kubernetes.apiserver.request.code","kubernetes.apiserver.request.content_type","kubernetes.apiserver.request.dry_run","kubernetes.apiserver.request.kind","kubernetes.apiserver.request.component","kubernetes.apiserver.request.group","kubernetes.apiserver.request.version","kubernetes.apiserver.request.handler","kubernetes.apiserver.request.method","kubernetes.apiserver.request.host","kubernetes.controllermanager.handler","kubernetes.controllermanager.code","kubernetes.controllermanager.method","kubernetes.controllermanager.host","kubernetes.controllermanager.name","kubernetes.controllermanager.zone","kubernetes.event.message","kubernetes.event.reason","kubernetes.event.type","kubernetes.event.source.component","kubernetes.event.source.host","kubernetes.event.metadata.generate_name","kubernetes.event.metadata.name","kubernetes.event.metadata.namespace","kubernetes.event.metadata.resource_version","kubernetes.event.metadata.uid","kubernetes.event.metadata.self_link","kubernetes.event.involved_object.api_version","kubernetes.event.involved_object.kind","kubernetes.event.involved_object.name","kubernetes.event.involved_object.resource_version","kubernetes.event.involved_object.uid","kubernetes.proxy.handler","kubernetes.proxy.code","kubernetes.proxy.method","kubernetes.proxy.host","kubernetes.scheduler.handler","kubernetes.scheduler.code","kubernetes.scheduler.method","kubernetes.scheduler.host","kubernetes.scheduler.name","kubernetes.scheduler.result","kubernetes.scheduler.operation","kubernetes.container.id","kubernetes.container.status.phase","kubernetes.container.status.reason","kubernetes.cronjob.name","kubernetes.cronjob.schedule","kubernetes.cronjob.concurrency","kubernetes.daemonset.name","kubernetes.node.status.ready","kubernetes.node.status.memory_pressure","kubernetes.node.status.disk_pressure","kubernetes.node.status.out_of_disk","kubernetes.node.status.pid_pressure","kubernetes.persistentvolume.name","kubernetes.persistentvolume.phase","kubernetes.persistentvolume.storage_class","kubernetes.persistentvolumeclaim.name","kubernetes.persistentvolumeclaim.volume_name","kubernetes.persistentvolumeclaim.phase","kubernetes.persistentvolumeclaim.access_mode","kubernetes.persistentvolumeclaim.storage_class","kubernetes.pod.status.phase","kubernetes.pod.status.ready","kubernetes.pod.status.scheduled","kubernetes.resourcequota.name","kubernetes.resourcequota.type","kubernetes.resourcequota.resource","kubernetes.service.name","kubernetes.service.cluster_ip","kubernetes.service.external_name","kubernetes.service.external_ip","kubernetes.service.load_balancer_ip","kubernetes.service.type","kubernetes.service.ingress_ip","kubernetes.service.ingress_hostname","kubernetes.storageclass.name","kubernetes.storageclass.provisioner","kubernetes.storageclass.reclaim_policy","kubernetes.storageclass.volume_binding_mode","kubernetes.system.container","kubernetes.volume.name","kvm.name","kvm.dommemstat.stat.name","kvm.dommemstat.name","kvm.status.state","logstash.node.state.pipeline.id","logstash.node.state.pipeline.hash","logstash.node.jvm.version","logstash.node.stats.logstash.uuid","logstash.node.stats.logstash.version","logstash.node.stats.pipelines.id","logstash.node.stats.pipelines.hash","logstash.node.stats.pipelines.queue.type","logstash.node.stats.pipelines.vertices.pipeline_ephemeral_id","logstash.node.stats.pipelines.vertices.id","mongodb.collstats.db","mongodb.collstats.collection","mongodb.collstats.name","mongodb.dbstats.db","mongodb.metrics.replication.executor.network_interface","mongodb.replstatus.set_name","mongodb.replstatus.members.primary.host","mongodb.replstatus.members.primary.optime","mongodb.replstatus.members.secondary.hosts","mongodb.replstatus.members.secondary.optimes","mongodb.replstatus.members.recovering.hosts","mongodb.replstatus.members.unknown.hosts","mongodb.replstatus.members.startup2.hosts","mongodb.replstatus.members.arbiter.hosts","mongodb.replstatus.members.down.hosts","mongodb.replstatus.members.rollback.hosts","mongodb.replstatus.members.unhealthy.hosts","mongodb.status.storage_engine.name","munin.plugin.name","mysql.galera_status.cluster.status","mysql.galera_status.connected","mysql.galera_status.evs.evict","mysql.galera_status.evs.state","mysql.galera_status.local.state","mysql.galera_status.ready","mysql.performance.events_statements.digest","mysql.performance.table_io_waits.object.schema","mysql.performance.table_io_waits.object.name","mysql.performance.table_io_waits.index.name","nats.server.id","nats.connection.name","nats.route.remote_id","nginx.stubstatus.hostname","php_fpm.pool.name","php_fpm.pool.process_manager","php_fpm.process.state","php_fpm.process.script","postgresql.activity.database.name","postgresql.activity.user.name","postgresql.activity.application_name","postgresql.activity.client.address","postgresql.activity.client.hostname","postgresql.activity.backend_type","postgresql.activity.state","postgresql.activity.query","postgresql.activity.wait_event","postgresql.activity.wait_event_type","postgresql.database.name","postgresql.statement.query.text","rabbitmq.vhost","rabbitmq.connection.name","rabbitmq.connection.state","rabbitmq.connection.type","rabbitmq.connection.host","rabbitmq.connection.peer.host","rabbitmq.connection.client_provided.name","rabbitmq.exchange.name","rabbitmq.node.name","rabbitmq.node.type","rabbitmq.queue.name","rabbitmq.queue.state","redis.info.memory.max.policy","redis.info.memory.allocator","redis.info.persistence.rdb.bgsave.last_status","redis.info.persistence.aof.bgrewrite.last_status","redis.info.persistence.aof.write.last_status","redis.info.replication.role","redis.info.replication.master.link_status","redis.info.server.git_sha1","redis.info.server.git_dirty","redis.info.server.build_id","redis.info.server.mode","redis.info.server.arch_bits","redis.info.server.multiplexing_api","redis.info.server.gcc_version","redis.info.server.run_id","redis.info.server.config_file","redis.key.name","redis.key.id","redis.key.type","redis.keyspace.id","process.state","system.diskio.name","system.diskio.serial_number","system.filesystem.device_name","system.filesystem.type","system.filesystem.mount_point","system.network.name","system.process.state","system.process.cmdline","system.process.cgroup.id","system.process.cgroup.path","system.process.cgroup.cpu.id","system.process.cgroup.cpu.path","system.process.cgroup.cpuacct.id","system.process.cgroup.cpuacct.path","system.process.cgroup.memory.id","system.process.cgroup.memory.path","system.process.cgroup.blkio.id","system.process.cgroup.blkio.path","system.raid.name","system.raid.status","system.raid.level","system.raid.sync_action","system.service.name","system.service.load_state","system.service.state","system.service.sub_state","system.service.exec_code","system.service.unit_file.state","system.service.unit_file.vendor_preset","system.socket.remote.host","system.socket.remote.etld_plus_one","system.socket.remote.host_error","system.socket.process.cmdline","system.users.id","system.users.seat","system.users.path","system.users.type","system.users.service","system.users.state","system.users.scope","system.users.remote_host","uwsgi.status.worker.status","vsphere.datastore.name","vsphere.datastore.fstype","vsphere.host.name","vsphere.host.network_names","vsphere.virtualmachine.host.id","vsphere.virtualmachine.host.hostname","vsphere.virtualmachine.name","vsphere.virtualmachine.os","vsphere.virtualmachine.network_names","windows.perfmon.instance","windows.service.id","windows.service.name","windows.service.display_name","windows.service.start_type","windows.service.start_name","windows.service.path_name","windows.service.state","windows.service.exit_code","zookeeper.mntr.hostname","zookeeper.mntr.server_state","zookeeper.server.mode","zookeeper.server.zxid","fields.*"]},"refresh_interval":"5s","routing":{"allocation":{"include":{"_tier_preference":"data_content"}}}}}
[00:13:56]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-kibana-6-2017.10.05] creating index, cause [api], templates [], shards [1]/[1]
[00:13:56]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-kibana-6-2017.10.05"
[00:13:56]                 │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-kibana-6-2017.10.05" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:13:56]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-alerts-6] creating index, cause [api], templates [], shards [1]/[1]
[00:13:56]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-alerts-6"
[00:13:56]                 │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-alerts-6" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:13:56]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-logstash-6-2017.10.05] creating index, cause [api], templates [], shards [1]/[1]
[00:13:56]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-logstash-6-2017.10.05"
[00:13:56]                 │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-logstash-6-2017.10.05" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:13:56]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-es-6-2017.10.05] creating index, cause [api], templates [], shards [1]/[1]
[00:13:56]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-es-6-2017.10.05"
[00:13:56]                 │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-es-6-2017.10.05" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:13:56]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/MgLVNx48T7-Htr-zhajqRw] update_mapping [_doc]
[00:13:56]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/MgLVNx48T7-Htr-zhajqRw] update_mapping [_doc]
[00:13:56]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/MgLVNx48T7-Htr-zhajqRw] update_mapping [_doc]
[00:13:57]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/MgLVNx48T7-Htr-zhajqRw] update_mapping [_doc]
[00:13:57]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/MgLVNx48T7-Htr-zhajqRw] update_mapping [_doc]
[00:13:58]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Indexed 2930 docs into "metricbeat-8.0.0"
[00:13:58]                 │ debg replacing kibana config doc: {}
[00:13:58]                 │ debg creating user basic_monitoring_user
[00:13:58]                 │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] added user [basic_monitoring_user]
[00:13:58]                 │ debg created user basic_monitoring_user
[00:13:58]                 │ debg SecurityPage.forceLogout
[00:13:58]                 │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=100
[00:13:58]                 │ debg Already on the login page, not forcing anything
[00:13:58]                 │ debg TestSubjects.setValue(loginUsername, basic_monitoring_user)
[00:13:58]                 │ debg TestSubjects.click(loginUsername)
[00:13:58]                 │ debg Find.clickByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:13:58]                 │ debg Find.findByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:13:59]                 │ debg TestSubjects.setValue(loginPassword, monitoring_user_password)
[00:13:59]                 │ debg TestSubjects.click(loginPassword)
[00:13:59]                 │ debg Find.clickByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:13:59]                 │ debg Find.findByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:13:59]                 │ debg TestSubjects.click(loginSubmit)
[00:13:59]                 │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:13:59]                 │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:13:59]                 │ debg Find.waitForDeletedByCssSelector('.kibanaWelcomeLogo') with timeout=10000
[00:13:59]                 │ proc [kibana]   log   [11:55:57.759] [info][plugins][routes][security] Logging in with provider "basic" (basic)
[00:13:59]                 │ debg browser[INFO] http://localhost:61131/app/home 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:13:59]                 │
[00:13:59]                 │ debg browser[INFO] http://localhost:61131/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:13:59]                 │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:14:01]                 │ debg navigating to monitoring url: http://localhost:61131/app/monitoring
[00:14:01]                 │ debg navigate to: http://localhost:61131/app/monitoring
[00:14:01]                 │ debg browser[INFO] http://localhost:61131/app/monitoring?_t=1617623759421 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:14:01]                 │
[00:14:01]                 │ debg browser[INFO] http://localhost:61131/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:14:01]                 │ debg ... sleep(700) start
[00:14:02]                 │ debg ... sleep(700) end
[00:14:02]                 │ debg returned from get, calling refresh
[00:14:02]                 │ debg browser[INFO] http://localhost:61131/app/monitoring?_t=1617623759421#/no-data?_g=(filters:!(),refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now)) 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:14:02]                 │
[00:14:02]                 │ debg browser[INFO] http://localhost:61131/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:14:03]                 │ debg currentUrl = http://localhost:61131/app/monitoring#/no-data?_g=(filters:!(),refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))
[00:14:03]                 │          appUrl = http://localhost:61131/app/monitoring
[00:14:03]                 │ debg TestSubjects.find(kibanaChrome)
[00:14:03]                 │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:14:03]                 │ debg browser[WARNING] http://localhost:61131/41829/bundles/plugin/monitoring/monitoring.chunk.1.js 29:205493 "\"#noDataReact\" element has not been added to the DOM yet"
[00:14:03]                 │ debg ... sleep(501) start
[00:14:03]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:03]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:03]                 │ debg ... sleep(501) end
[00:14:03]                 │ debg in navigateTo url = http://localhost:61131/app/monitoring#/no-data?_g=(filters:!(),refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))
[00:14:03]                 │ debg TestSubjects.exists(statusPageContainer)
[00:14:03]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:14:06]                 │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:14:06]                 │ debg pauseAutoRefresh
[00:14:06]                 │ debg setting menu open state [name=QuickSelectTime Menu] [state=open]
[00:14:06]                 │ debg TestSubjects.exists(superDatePickerQuickMenu)
[00:14:06]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=1000
[00:14:08]                 │ debg --- retry.tryForTime error: [data-test-subj="superDatePickerQuickMenu"] is not displayed
[00:14:08]                 │ debg TestSubjects.click(superDatePickerToggleQuickMenuButton)
[00:14:08]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:14:08]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:14:08]                 │ debg TestSubjects.exists(superDatePickerQuickMenu)
[00:14:08]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=10000
[00:14:08]                 │ debg TestSubjects.getAttribute(superDatePickerRefreshIntervalInput, value, tryTimeout=120000, findTimeout=10000)
[00:14:08]                 │ debg TestSubjects.find(superDatePickerRefreshIntervalInput)
[00:14:08]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerRefreshIntervalInput"]') with timeout=10000
[00:14:08]                 │ debg TestSubjects.find(superDatePickerRefreshIntervalUnitsSelect)
[00:14:08]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerRefreshIntervalUnitsSelect"]') with timeout=10000
[00:14:08]                 │ debg Find.allDescendantDisplayedByCssSelector('option')
[00:14:08]                 │ debg TestSubjects.getVisibleText(superDatePickerToggleRefreshButton)
[00:14:08]                 │ debg TestSubjects.find(superDatePickerToggleRefreshButton)
[00:14:08]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleRefreshButton"]') with timeout=10000
[00:14:08]                 │ debg pause auto refresh
[00:14:08]                 │ debg TestSubjects.click(superDatePickerToggleRefreshButton)
[00:14:08]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerToggleRefreshButton"]') with timeout=10000
[00:14:08]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleRefreshButton"]') with timeout=10000
[00:14:09]                 │ debg setting menu open state [name=QuickSelectTime Menu] [state=closed]
[00:14:09]                 │ debg TestSubjects.exists(superDatePickerQuickMenu)
[00:14:09]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=1000
[00:14:09]                 │ debg TestSubjects.click(superDatePickerToggleQuickMenuButton)
[00:14:09]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:14:09]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:14:09]                 │ debg Find.waitForDeletedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=10000
[00:14:10]                 │ debg Setting absolute range to Oct 5, 2017 @ 20:31:48.354 to Oct 5, 2017 @ 20:35:12.176
[00:14:10]                 │ debg TestSubjects.exists(superDatePickerToggleQuickMenuButton)
[00:14:10]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=20000
[00:14:10]                 │ debg TestSubjects.exists(superDatePickerShowDatesButton)
[00:14:10]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerShowDatesButton"]') with timeout=2500
[00:14:10]                 │ debg TestSubjects.click(superDatePickerShowDatesButton)
[00:14:10]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerShowDatesButton"]') with timeout=10000
[00:14:10]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerShowDatesButton"]') with timeout=10000
[00:14:10]                 │ debg TestSubjects.exists(superDatePickerstartDatePopoverButton)
[00:14:10]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerstartDatePopoverButton"]') with timeout=2500
[00:14:10]                 │ debg TestSubjects.click(superDatePickerendDatePopoverButton)
[00:14:10]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerendDatePopoverButton"]') with timeout=10000
[00:14:10]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerendDatePopoverButton"]') with timeout=10000
[00:14:10]                 │ debg Find.findByCssSelector('div.euiPopover__panel-isOpen') with timeout=10000
[00:14:10]                 │ debg TestSubjects.click(superDatePickerAbsoluteTab)
[00:14:10]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:14:10]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:14:10]                 │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:14:10]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:10]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:10]                 │ debg TestSubjects.setValue(superDatePickerAbsoluteDateInput, Oct 5, 2017 @ 20:35:12.176)
[00:14:10]                 │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:14:10]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:10]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:11]                 │ debg TestSubjects.click(superDatePickerstartDatePopoverButton)
[00:14:11]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerstartDatePopoverButton"]') with timeout=10000
[00:14:11]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerstartDatePopoverButton"]') with timeout=10000
[00:14:11]                 │ debg Find.waitForElementStale with timeout=10000
[00:14:11]                 │ debg Find.findByCssSelector('div.euiPopover__panel-isOpen') with timeout=10000
[00:14:11]                 │ debg TestSubjects.click(superDatePickerAbsoluteTab)
[00:14:11]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:14:11]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:14:11]                 │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:14:11]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:11]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:11]                 │ debg TestSubjects.setValue(superDatePickerAbsoluteDateInput, Oct 5, 2017 @ 20:31:48.354)
[00:14:11]                 │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:14:11]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:11]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:11]                 │ debg TestSubjects.exists(superDatePickerApplyTimeButton)
[00:14:11]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=2500
[00:14:14]                 │ debg --- retry.tryForTime error: [data-test-subj="superDatePickerApplyTimeButton"] is not displayed
[00:14:14]                 │ debg TestSubjects.click(querySubmitButton)
[00:14:14]                 │ debg Find.clickByCssSelector('[data-test-subj="querySubmitButton"]') with timeout=10000
[00:14:14]                 │ debg Find.findByCssSelector('[data-test-subj="querySubmitButton"]') with timeout=10000
[00:14:15]                 │ debg Find.waitForElementStale with timeout=10000
[00:14:15]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:15]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:15]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:15]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:15]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:15]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:15]                 │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:14:15]                 │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:14:15]                 │ debg TestSubjects.click(clusterItemContainerElasticsearch > esNumberOfIndices)
[00:14:15]                 │ debg Find.clickByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:14:15]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:14:25]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:14:25]                 │      Wait timed out after 10007ms
[00:14:25]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:14:35]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:14:35]                 │      Wait timed out after 10006ms
[00:14:36]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:14:46]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:14:46]                 │      Wait timed out after 10008ms
[00:14:46]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:14:56]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:14:56]                 │      Wait timed out after 10032ms
[00:14:57]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:15:07]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:15:07]                 │      Wait timed out after 10007ms
[00:15:07]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:15:17]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:15:17]                 │      Wait timed out after 10032ms
[00:15:18]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:15:28]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:15:28]                 │      Wait timed out after 10014ms
[00:15:28]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:15:39]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:15:39]                 │      Wait timed out after 10058ms
[00:15:39]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:15:49]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:15:49]                 │      Wait timed out after 10023ms
[00:15:50]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:16:00]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:16:00]                 │      Wait timed out after 10049ms
[00:16:00]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:16:10]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:16:10]                 │      Wait timed out after 10023ms
[00:16:11]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:16:21]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:16:21]                 │      Wait timed out after 10022ms
[00:16:21]                 │ info Taking screenshot "/dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/screenshots/failure/Monitoring app Elasticsearch index detail mb Active Indices _before all_ hook for _should have an index summary with green status index with full shard allocation_.png"
[00:16:21]                 │ info Current URL is: http://localhost:61131/app/monitoring#/no-data?_g=(filters:!(),refreshInterval:(pause:!t,value:10000),time:(from:%272017-10-05T20:31:48.354Z%27,to:%272017-10-05T20:35:12.176Z%27))
[00:16:21]                 │ info Saving page source to: /dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/failure_debug/html/Monitoring app Elasticsearch index detail mb Active Indices _before all_ hook for _should have an index summary with green status index with full shard allocation_.html
[00:16:21]                 └- ✖ fail: Monitoring app Elasticsearch index detail mb Active Indices "before all" hook for "should have an index summary with green status index with full shard allocation"
[00:16:21]                 │      Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:16:21]                 │ Wait timed out after 10022ms
[00:16:21]                 │     at /dev/shm/workspace/parallel/13/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
[00:16:21]                 │     at runMicrotasks (<anonymous>)
[00:16:21]                 │     at processTicksAndRejections (internal/process/task_queues.js:93:5)
[00:16:21]                 │       at onFailure (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:17:9)
[00:16:21]                 │       at retryForSuccess (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:57:13)
[00:16:21]                 │       at Retry.try (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry.ts:32:14)
[00:16:21]                 │       at Proxy.clickByCssSelector (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/find.ts:420:7)
[00:16:21]                 │       at TestSubjects.click (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/test_subjects.ts:109:7)
[00:16:21]                 │       at Context.<anonymous> (test/functional/apps/monitoring/elasticsearch/index_detail_mb.js:33:9)
[00:16:21]                 │       at Object.apply (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)
[00:16:21]                 │ 
[00:16:21]                 │ 

Stack Trace

Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
Wait timed out after 10022ms
    at /dev/shm/workspace/parallel/13/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at onFailure (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:17:9)
    at retryForSuccess (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:57:13)
    at Retry.try (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry.ts:32:14)
    at Proxy.clickByCssSelector (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/find.ts:420:7)
    at TestSubjects.click (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/test_subjects.ts:109:7)
    at Context.<anonymous> (test/functional/apps/monitoring/elasticsearch/index_detail_mb.js:33:9)
    at Object.apply (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)

---

Kibana Pipeline / general / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/monitoring/elasticsearch/overview_mb·js.Monitoring app Elasticsearch overview mb "before all" hook for "should have an Elasticsearch Cluster Summary Status with correct info"

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: Monitoring app
[00:00:00]           └-> "before all" hook in "Monitoring app"
[00:05:18]           └-: Elasticsearch overview mb
[00:05:18]             └-> "before all" hook for "should have an Elasticsearch Cluster Summary Status with correct info"
[00:05:18]             └-> "before all" hook for "should have an Elasticsearch Cluster Summary Status with correct info"
[00:05:18]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Loading "mappings.json"
[00:05:18]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Loading "data.json.gz"
[00:05:18]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0] creating index, cause [api], templates [], shards [1]/[1]
[00:05:18]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index "metricbeat-8.0.0"
[00:05:18]               │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] "metricbeat-8.0.0" settings {"index":{"codec":"best_compression","lifecycle":{"name":"metricbeat","rollover_alias":"metricbeat-8.0.0"},"mapping":{"total_fields":{"limit":"10000"}},"max_docvalue_fields_search":"200","number_of_replicas":"1","number_of_shards":"1","query":{"default_field":["message","tags","agent.ephemeral_id","agent.id","agent.name","agent.type","agent.version","as.organization.name","client.address","client.as.organization.name","client.domain","client.geo.city_name","client.geo.continent_name","client.geo.country_iso_code","client.geo.country_name","client.geo.name","client.geo.region_iso_code","client.geo.region_name","client.mac","client.registered_domain","client.top_level_domain","client.user.domain","client.user.email","client.user.full_name","client.user.group.domain","client.user.group.id","client.user.group.name","client.user.hash","client.user.id","client.user.name","cloud.account.id","cloud.availability_zone","cloud.instance.id","cloud.instance.name","cloud.machine.type","cloud.provider","cloud.region","container.id","container.image.name","container.image.tag","container.name","container.runtime","destination.address","destination.as.organization.name","destination.domain","destination.geo.city_name","destination.geo.continent_name","destination.geo.country_iso_code","destination.geo.country_name","destination.geo.name","destination.geo.region_iso_code","destination.geo.region_name","destination.mac","destination.registered_domain","destination.top_level_domain","destination.user.domain","destination.user.email","destination.user.full_name","destination.user.group.domain","destination.user.group.id","destination.user.group.name","destination.user.hash","destination.user.id","destination.user.name","dns.answers.class","dns.answers.data","dns.answers.name","dns.answers.type","dns.header_flags","dns.id","dns.op_code","dns.question.class","dns.question.name","dns.question.registered_domain","dns.question.subdomain","dns.question.top_level_domain","dns.question.type","dns.response_code","dns.type","ecs.version","error.code","error.id","error.message","error.stack_trace","error.type","event.action","event.category","event.code","event.dataset","event.hash","event.id","event.kind","event.module","event.original","event.outcome","event.provider","event.timezone","event.type","file.device","file.directory","file.extension","file.gid","file.group","file.hash.md5","file.hash.sha1","file.hash.sha256","file.hash.sha512","file.inode","file.mode","file.name","file.owner","file.path","file.target_path","file.type","file.uid","geo.city_name","geo.continent_name","geo.country_iso_code","geo.country_name","geo.name","geo.region_iso_code","geo.region_name","group.domain","group.id","group.name","hash.md5","hash.sha1","hash.sha256","hash.sha512","host.architecture","host.geo.city_name","host.geo.continent_name","host.geo.country_iso_code","host.geo.country_name","host.geo.name","host.geo.region_iso_code","host.geo.region_name","host.hostname","host.id","host.mac","host.name","host.os.family","host.os.full","host.os.kernel","host.os.name","host.os.platform","host.os.version","host.type","host.user.domain","host.user.email","host.user.full_name","host.user.group.domain","host.user.group.id","host.user.group.name","host.user.hash","host.user.id","host.user.name","http.request.body.content","http.request.method","http.request.referrer","http.response.body.content","http.version","log.level","log.logger","log.origin.file.name","log.origin.function","log.original","log.syslog.facility.name","log.syslog.severity.name","network.application","network.community_id","network.direction","network.iana_number","network.name","network.protocol","network.transport","network.type","observer.geo.city_name","observer.geo.continent_name","observer.geo.country_iso_code","observer.geo.country_name","observer.geo.name","observer.geo.region_iso_code","observer.geo.region_name","observer.hostname","observer.mac","observer.name","observer.os.family","observer.os.full","observer.os.kernel","observer.os.name","observer.os.platform","observer.os.version","observer.product","observer.serial_number","observer.type","observer.vendor","observer.version","organization.id","organization.name","os.family","os.full","os.kernel","os.name","os.platform","os.version","package.architecture","package.checksum","package.description","package.install_scope","package.license","package.name","package.path","package.version","process.args","text","process.executable","process.hash.md5","process.hash.sha1","process.hash.sha256","process.hash.sha512","process.name","text","text","text","text","text","process.thread.name","process.title","process.working_directory","server.address","server.as.organization.name","server.domain","server.geo.city_name","server.geo.continent_name","server.geo.country_iso_code","server.geo.country_name","server.geo.name","server.geo.region_iso_code","server.geo.region_name","server.mac","server.registered_domain","server.top_level_domain","server.user.domain","server.user.email","server.user.full_name","server.user.group.domain","server.user.group.id","server.user.group.name","server.user.hash","server.user.id","server.user.name","service.ephemeral_id","service.id","service.name","service.node.name","service.state","service.type","service.version","source.address","source.as.organization.name","source.domain","source.geo.city_name","source.geo.continent_name","source.geo.country_iso_code","source.geo.country_name","source.geo.name","source.geo.region_iso_code","source.geo.region_name","source.mac","source.registered_domain","source.top_level_domain","source.user.domain","source.user.email","source.user.full_name","source.user.group.domain","source.user.group.id","source.user.group.name","source.user.hash","source.user.id","source.user.name","threat.framework","threat.tactic.id","threat.tactic.name","threat.tactic.reference","threat.technique.id","threat.technique.name","threat.technique.reference","tracing.trace.id","tracing.transaction.id","url.domain","url.extension","url.fragment","url.full","url.original","url.password","url.path","url.query","url.registered_domain","url.scheme","url.top_level_domain","url.username","user.domain","user.email","user.full_name","user.group.domain","user.group.id","user.group.name","user.hash","user.id","user.name","user_agent.device.name","user_agent.name","text","user_agent.original","user_agent.os.family","user_agent.os.full","user_agent.os.kernel","user_agent.os.name","user_agent.os.platform","user_agent.os.version","user_agent.version","text","agent.hostname","timeseries.instance","cloud.project.id","cloud.image.id","host.os.build","host.os.codename","kubernetes.pod.name","kubernetes.pod.uid","kubernetes.namespace","kubernetes.node.name","kubernetes.replicaset.name","kubernetes.deployment.name","kubernetes.statefulset.name","kubernetes.container.name","kubernetes.container.image","jolokia.agent.version","jolokia.agent.id","jolokia.server.product","jolokia.server.version","jolokia.server.vendor","jolokia.url","metricset.name","service.address","service.hostname","type","systemd.fragment_path","systemd.unit","aerospike.namespace.name","aerospike.namespace.node.host","aerospike.namespace.node.name","apache.status.hostname","beat.id","beat.type","beat.state.service.id","beat.state.service.name","beat.state.service.version","beat.state.input.names","beat.state.beat.host","beat.state.beat.name","beat.state.beat.type","beat.state.beat.uuid","beat.state.beat.version","beat.state.cluster.uuid","beat.state.host.containerized","beat.state.host.os.kernel","beat.state.host.os.name","beat.state.host.os.platform","beat.state.host.os.version","beat.state.module.names","beat.state.output.name","beat.state.queue.name","beat.stats.beat.name","beat.stats.beat.host","beat.stats.beat.type","beat.stats.beat.uuid","beat.stats.beat.version","beat.stats.info.ephemeral_id","beat.stats.libbeat.output.type","ceph.cluster_health.overall_status","ceph.cluster_health.timechecks.round.status","ceph.mgr_osd_pool_stats.pool_name","ceph.monitor_health.health","ceph.monitor_health.name","ceph.osd_df.name","ceph.osd_df.device_class","ceph.osd_tree.name","ceph.osd_tree.type","ceph.osd_tree.children","ceph.osd_tree.status","ceph.osd_tree.device_class","ceph.osd_tree.father","ceph.pool_disk.name","couchbase.bucket.name","couchbase.bucket.type","couchbase.node.hostname","docker.container.command","docker.container.status","docker.container.tags","docker.event.status","docker.event.id","docker.event.from","docker.event.type","docker.event.action","docker.event.actor.id","docker.healthcheck.status","docker.healthcheck.event.output","docker.image.id.current","docker.image.id.parent","docker.image.tags","docker.info.id","docker.network.interface","elasticsearch.cluster.name","elasticsearch.cluster.id","elasticsearch.cluster.state.id","elasticsearch.node.id","elasticsearch.node.name","elasticsearch.ccr.remote_cluster","elasticsearch.ccr.leader.index","elasticsearch.ccr.follower.index","elasticsearch.cluster.stats.version","elasticsearch.cluster.stats.state.nodes_hash","elasticsearch.cluster.stats.state.master_node","elasticsearch.cluster.stats.state.version","elasticsearch.cluster.stats.state.state_uuid","elasticsearch.cluster.stats.status","elasticsearch.cluster.stats.license.status","elasticsearch.cluster.stats.license.type","elasticsearch.enrich.executing_policy.name","elasticsearch.enrich.executing_policy.task.task","elasticsearch.enrich.executing_policy.task.action","elasticsearch.enrich.executing_policy.task.parent_task_id","elasticsearch.index.uuid","elasticsearch.index.status","elasticsearch.index.name","elasticsearch.index.recovery.index.files.percent","elasticsearch.index.recovery.name","elasticsearch.index.recovery.type","elasticsearch.index.recovery.stage","elasticsearch.index.recovery.translog.percent","elasticsearch.index.recovery.target.transport_address","elasticsearch.index.recovery.target.id","elasticsearch.index.recovery.target.host","elasticsearch.index.recovery.target.name","elasticsearch.index.recovery.source.transport_address","elasticsearch.index.recovery.source.id","elasticsearch.index.recovery.source.host","elasticsearch.index.recovery.source.name","elasticsearch.ml.job.id","elasticsearch.ml.job.state","elasticsearch.ml.job.model_size.memory_status","elasticsearch.node.version","elasticsearch.node.jvm.version","elasticsearch.node.stats.os.cgroup.memory.control_group","elasticsearch.cluster.pending_task.source","elasticsearch.shard.state","elasticsearch.shard.relocating_node.name","elasticsearch.shard.relocating_node.id","elasticsearch.shard.source_node.name","elasticsearch.shard.source_node.uuid","etcd.api_version","etcd.leader.leader","etcd.self.id","etcd.self.leaderinfo.leader","etcd.self.leaderinfo.starttime","etcd.self.leaderinfo.uptime","etcd.self.name","etcd.self.starttime","etcd.self.state","golang.expvar.cmdline","golang.heap.cmdline","graphite.server.example","haproxy.stat.status","haproxy.stat.service_name","haproxy.stat.cookie","haproxy.stat.load_balancing_algorithm","haproxy.stat.check.status","haproxy.stat.check.health.last","haproxy.stat.proxy.name","haproxy.stat.proxy.mode","haproxy.stat.agent.status","haproxy.stat.agent.description","haproxy.stat.agent.check.description","haproxy.stat.source.address","http.response.code","http.response.phrase","kafka.broker.address","kafka.topic.name","kafka.partition.topic_id","kafka.partition.topic_broker_id","kafka.broker.mbean","kafka.consumer.mbean","kafka.consumergroup.broker.address","kafka.consumergroup.id","kafka.consumergroup.topic","kafka.consumergroup.meta","kafka.consumergroup.client.id","kafka.consumergroup.client.host","kafka.consumergroup.client.member_id","kafka.partition.topic.name","kafka.partition.broker.address","kafka.producer.mbean","kibana.settings.uuid","kibana.settings.name","kibana.settings.index","kibana.settings.host","kibana.settings.transport_address","kibana.settings.version","kibana.settings.status","kibana.settings.locale","kibana.stats.kibana.status","kibana.stats.usage.index","kibana.stats.name","kibana.stats.index","kibana.stats.host.name","kibana.stats.status","kibana.stats.os.distro","kibana.stats.os.distroRelease","kibana.stats.os.platform","kibana.stats.os.platformRelease","kibana.status.name","kibana.status.status.overall.state","kubernetes.apiserver.request.client","kubernetes.apiserver.request.resource","kubernetes.apiserver.request.subresource","kubernetes.apiserver.request.scope","kubernetes.apiserver.request.verb","kubernetes.apiserver.request.code","kubernetes.apiserver.request.content_type","kubernetes.apiserver.request.dry_run","kubernetes.apiserver.request.kind","kubernetes.apiserver.request.component","kubernetes.apiserver.request.group","kubernetes.apiserver.request.version","kubernetes.apiserver.request.handler","kubernetes.apiserver.request.method","kubernetes.apiserver.request.host","kubernetes.controllermanager.handler","kubernetes.controllermanager.code","kubernetes.controllermanager.method","kubernetes.controllermanager.host","kubernetes.controllermanager.name","kubernetes.controllermanager.zone","kubernetes.event.message","kubernetes.event.reason","kubernetes.event.type","kubernetes.event.source.component","kubernetes.event.source.host","kubernetes.event.metadata.generate_name","kubernetes.event.metadata.name","kubernetes.event.metadata.namespace","kubernetes.event.metadata.resource_version","kubernetes.event.metadata.uid","kubernetes.event.metadata.self_link","kubernetes.event.involved_object.api_version","kubernetes.event.involved_object.kind","kubernetes.event.involved_object.name","kubernetes.event.involved_object.resource_version","kubernetes.event.involved_object.uid","kubernetes.proxy.handler","kubernetes.proxy.code","kubernetes.proxy.method","kubernetes.proxy.host","kubernetes.scheduler.handler","kubernetes.scheduler.code","kubernetes.scheduler.method","kubernetes.scheduler.host","kubernetes.scheduler.name","kubernetes.scheduler.result","kubernetes.scheduler.operation","kubernetes.container.id","kubernetes.container.status.phase","kubernetes.container.status.reason","kubernetes.cronjob.name","kubernetes.cronjob.schedule","kubernetes.cronjob.concurrency","kubernetes.daemonset.name","kubernetes.node.status.ready","kubernetes.node.status.memory_pressure","kubernetes.node.status.disk_pressure","kubernetes.node.status.out_of_disk","kubernetes.node.status.pid_pressure","kubernetes.persistentvolume.name","kubernetes.persistentvolume.phase","kubernetes.persistentvolume.storage_class","kubernetes.persistentvolumeclaim.name","kubernetes.persistentvolumeclaim.volume_name","kubernetes.persistentvolumeclaim.phase","kubernetes.persistentvolumeclaim.access_mode","kubernetes.persistentvolumeclaim.storage_class","kubernetes.pod.status.phase","kubernetes.pod.status.ready","kubernetes.pod.status.scheduled","kubernetes.resourcequota.name","kubernetes.resourcequota.type","kubernetes.resourcequota.resource","kubernetes.service.name","kubernetes.service.cluster_ip","kubernetes.service.external_name","kubernetes.service.external_ip","kubernetes.service.load_balancer_ip","kubernetes.service.type","kubernetes.service.ingress_ip","kubernetes.service.ingress_hostname","kubernetes.storageclass.name","kubernetes.storageclass.provisioner","kubernetes.storageclass.reclaim_policy","kubernetes.storageclass.volume_binding_mode","kubernetes.system.container","kubernetes.volume.name","kvm.name","kvm.dommemstat.stat.name","kvm.dommemstat.name","kvm.status.state","logstash.node.state.pipeline.id","logstash.node.state.pipeline.hash","logstash.node.jvm.version","logstash.node.stats.logstash.uuid","logstash.node.stats.logstash.version","logstash.node.stats.pipelines.id","logstash.node.stats.pipelines.hash","logstash.node.stats.pipelines.queue.type","logstash.node.stats.pipelines.vertices.pipeline_ephemeral_id","logstash.node.stats.pipelines.vertices.id","mongodb.collstats.db","mongodb.collstats.collection","mongodb.collstats.name","mongodb.dbstats.db","mongodb.metrics.replication.executor.network_interface","mongodb.replstatus.set_name","mongodb.replstatus.members.primary.host","mongodb.replstatus.members.primary.optime","mongodb.replstatus.members.secondary.hosts","mongodb.replstatus.members.secondary.optimes","mongodb.replstatus.members.recovering.hosts","mongodb.replstatus.members.unknown.hosts","mongodb.replstatus.members.startup2.hosts","mongodb.replstatus.members.arbiter.hosts","mongodb.replstatus.members.down.hosts","mongodb.replstatus.members.rollback.hosts","mongodb.replstatus.members.unhealthy.hosts","mongodb.status.storage_engine.name","munin.plugin.name","mysql.galera_status.cluster.status","mysql.galera_status.connected","mysql.galera_status.evs.evict","mysql.galera_status.evs.state","mysql.galera_status.local.state","mysql.galera_status.ready","mysql.performance.events_statements.digest","mysql.performance.table_io_waits.object.schema","mysql.performance.table_io_waits.object.name","mysql.performance.table_io_waits.index.name","nats.server.id","nats.connection.name","nats.route.remote_id","nginx.stubstatus.hostname","php_fpm.pool.name","php_fpm.pool.process_manager","php_fpm.process.state","php_fpm.process.script","postgresql.activity.database.name","postgresql.activity.user.name","postgresql.activity.application_name","postgresql.activity.client.address","postgresql.activity.client.hostname","postgresql.activity.backend_type","postgresql.activity.state","postgresql.activity.query","postgresql.activity.wait_event","postgresql.activity.wait_event_type","postgresql.database.name","postgresql.statement.query.text","rabbitmq.vhost","rabbitmq.connection.name","rabbitmq.connection.state","rabbitmq.connection.type","rabbitmq.connection.host","rabbitmq.connection.peer.host","rabbitmq.connection.client_provided.name","rabbitmq.exchange.name","rabbitmq.node.name","rabbitmq.node.type","rabbitmq.queue.name","rabbitmq.queue.state","redis.info.memory.max.policy","redis.info.memory.allocator","redis.info.persistence.rdb.bgsave.last_status","redis.info.persistence.aof.bgrewrite.last_status","redis.info.persistence.aof.write.last_status","redis.info.replication.role","redis.info.replication.master.link_status","redis.info.server.git_sha1","redis.info.server.git_dirty","redis.info.server.build_id","redis.info.server.mode","redis.info.server.arch_bits","redis.info.server.multiplexing_api","redis.info.server.gcc_version","redis.info.server.run_id","redis.info.server.config_file","redis.key.name","redis.key.id","redis.key.type","redis.keyspace.id","process.state","system.diskio.name","system.diskio.serial_number","system.filesystem.device_name","system.filesystem.type","system.filesystem.mount_point","system.network.name","system.process.state","system.process.cmdline","system.process.cgroup.id","system.process.cgroup.path","system.process.cgroup.cpu.id","system.process.cgroup.cpu.path","system.process.cgroup.cpuacct.id","system.process.cgroup.cpuacct.path","system.process.cgroup.memory.id","system.process.cgroup.memory.path","system.process.cgroup.blkio.id","system.process.cgroup.blkio.path","system.raid.name","system.raid.status","system.raid.level","system.raid.sync_action","system.service.name","system.service.load_state","system.service.state","system.service.sub_state","system.service.exec_code","system.service.unit_file.state","system.service.unit_file.vendor_preset","system.socket.remote.host","system.socket.remote.etld_plus_one","system.socket.remote.host_error","system.socket.process.cmdline","system.users.id","system.users.seat","system.users.path","system.users.type","system.users.service","system.users.state","system.users.scope","system.users.remote_host","uwsgi.status.worker.status","vsphere.datastore.name","vsphere.datastore.fstype","vsphere.host.name","vsphere.host.network_names","vsphere.virtualmachine.host.id","vsphere.virtualmachine.host.hostname","vsphere.virtualmachine.name","vsphere.virtualmachine.os","vsphere.virtualmachine.network_names","windows.perfmon.instance","windows.service.id","windows.service.name","windows.service.display_name","windows.service.start_type","windows.service.start_name","windows.service.path_name","windows.service.state","windows.service.exit_code","zookeeper.mntr.hostname","zookeeper.mntr.server_state","zookeeper.server.mode","zookeeper.server.zxid","fields.*"]},"refresh_interval":"5s","routing":{"allocation":{"include":{"_tier_preference":"data_content"}}}}}
[00:05:18]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-kibana-6-2017.10.05] creating index, cause [api], templates [], shards [1]/[1]
[00:05:18]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-kibana-6-2017.10.05"
[00:05:18]               │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-kibana-6-2017.10.05" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:05:18]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-alerts-6] creating index, cause [api], templates [], shards [1]/[1]
[00:05:18]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-alerts-6"
[00:05:18]               │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-alerts-6" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:05:18]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-logstash-6-2017.10.05] creating index, cause [api], templates [], shards [1]/[1]
[00:05:18]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-logstash-6-2017.10.05"
[00:05:18]               │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-logstash-6-2017.10.05" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:05:18]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-es-6-2017.10.05] creating index, cause [api], templates [], shards [1]/[1]
[00:05:18]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-es-6-2017.10.05"
[00:05:18]               │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-es-6-2017.10.05" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:05:18]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/gR9eWDhARWKmVcAWUjiBdg] update_mapping [_doc]
[00:05:19]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/gR9eWDhARWKmVcAWUjiBdg] update_mapping [_doc]
[00:05:19]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/gR9eWDhARWKmVcAWUjiBdg] update_mapping [_doc]
[00:05:19]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/gR9eWDhARWKmVcAWUjiBdg] update_mapping [_doc]
[00:05:19]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/gR9eWDhARWKmVcAWUjiBdg] update_mapping [_doc]
[00:05:20]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Indexed 2930 docs into "metricbeat-8.0.0"
[00:05:20]               │ debg replacing kibana config doc: {}
[00:05:20]               │ debg creating user basic_monitoring_user
[00:05:20]               │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] added user [basic_monitoring_user]
[00:05:20]               │ debg created user basic_monitoring_user
[00:05:20]               │ debg SecurityPage.forceLogout
[00:05:20]               │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=100
[00:05:20]               │ debg Already on the login page, not forcing anything
[00:05:20]               │ debg TestSubjects.setValue(loginUsername, basic_monitoring_user)
[00:05:20]               │ debg TestSubjects.click(loginUsername)
[00:05:20]               │ debg Find.clickByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:05:20]               │ debg Find.findByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:05:21]               │ debg TestSubjects.setValue(loginPassword, monitoring_user_password)
[00:05:21]               │ debg TestSubjects.click(loginPassword)
[00:05:21]               │ debg Find.clickByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:05:21]               │ debg Find.findByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:05:21]               │ debg TestSubjects.click(loginSubmit)
[00:05:21]               │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:05:21]               │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:05:21]               │ debg Find.waitForDeletedByCssSelector('.kibanaWelcomeLogo') with timeout=10000
[00:05:21]               │ proc [kibana]   log   [11:38:01.442] [info][plugins][routes][security] Logging in with provider "basic" (basic)
[00:05:22]               │ debg browser[INFO] http://localhost:61131/app/home 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:05:22]               │
[00:05:22]               │ debg browser[INFO] http://localhost:61131/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:05:22]               │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:05:23]               │ debg navigating to monitoring url: http://localhost:61131/app/monitoring
[00:05:23]               │ debg navigate to: http://localhost:61131/app/monitoring
[00:05:24]               │ debg browser[INFO] http://localhost:61131/app/monitoring?_t=1617622683557 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:05:24]               │
[00:05:24]               │ debg browser[INFO] http://localhost:61131/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:05:24]               │ debg ... sleep(700) start
[00:05:25]               │ debg ... sleep(700) end
[00:05:25]               │ debg returned from get, calling refresh
[00:05:25]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:25]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:25]               │ debg browser[INFO] http://localhost:61131/app/monitoring?_t=1617622683557#/no-data?_g=(filters:!(),refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now)) 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:05:25]               │
[00:05:25]               │ debg browser[INFO] http://localhost:61131/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:05:25]               │ debg currentUrl = http://localhost:61131/app/monitoring#/no-data?_g=(filters:!(),refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))
[00:05:25]               │          appUrl = http://localhost:61131/app/monitoring
[00:05:25]               │ debg TestSubjects.find(kibanaChrome)
[00:05:25]               │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:05:26]               │ debg ... sleep(501) start
[00:05:26]               │ debg browser[WARNING] http://localhost:61131/41829/bundles/plugin/monitoring/monitoring.chunk.1.js 29:205493 "\"#noDataReact\" element has not been added to the DOM yet"
[00:05:26]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:26]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:27]               │ debg ... sleep(501) end
[00:05:27]               │ debg in navigateTo url = http://localhost:61131/app/monitoring#/no-data?_g=(filters:!(),refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))
[00:05:27]               │ debg TestSubjects.exists(statusPageContainer)
[00:05:27]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:05:29]               │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:05:30]               │ debg pauseAutoRefresh
[00:05:30]               │ debg setting menu open state [name=QuickSelectTime Menu] [state=open]
[00:05:30]               │ debg TestSubjects.exists(superDatePickerQuickMenu)
[00:05:30]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=1000
[00:05:31]               │ debg --- retry.tryForTime error: [data-test-subj="superDatePickerQuickMenu"] is not displayed
[00:05:31]               │ debg TestSubjects.click(superDatePickerToggleQuickMenuButton)
[00:05:31]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:05:31]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:05:31]               │ debg TestSubjects.exists(superDatePickerQuickMenu)
[00:05:31]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=10000
[00:05:32]               │ debg TestSubjects.getAttribute(superDatePickerRefreshIntervalInput, value, tryTimeout=120000, findTimeout=10000)
[00:05:32]               │ debg TestSubjects.find(superDatePickerRefreshIntervalInput)
[00:05:32]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerRefreshIntervalInput"]') with timeout=10000
[00:05:32]               │ debg TestSubjects.find(superDatePickerRefreshIntervalUnitsSelect)
[00:05:32]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerRefreshIntervalUnitsSelect"]') with timeout=10000
[00:05:32]               │ debg Find.allDescendantDisplayedByCssSelector('option')
[00:05:32]               │ debg TestSubjects.getVisibleText(superDatePickerToggleRefreshButton)
[00:05:32]               │ debg TestSubjects.find(superDatePickerToggleRefreshButton)
[00:05:32]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleRefreshButton"]') with timeout=10000
[00:05:32]               │ debg pause auto refresh
[00:05:32]               │ debg TestSubjects.click(superDatePickerToggleRefreshButton)
[00:05:32]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerToggleRefreshButton"]') with timeout=10000
[00:05:32]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleRefreshButton"]') with timeout=10000
[00:05:32]               │ debg setting menu open state [name=QuickSelectTime Menu] [state=closed]
[00:05:32]               │ debg TestSubjects.exists(superDatePickerQuickMenu)
[00:05:32]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=1000
[00:05:32]               │ debg TestSubjects.click(superDatePickerToggleQuickMenuButton)
[00:05:32]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:05:32]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:05:32]               │ debg Find.waitForDeletedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=10000
[00:05:33]               │ debg Setting absolute range to Oct 5, 2017 @ 20:31:48.354 to Oct 5, 2017 @ 20:35:12.176
[00:05:33]               │ debg TestSubjects.exists(superDatePickerToggleQuickMenuButton)
[00:05:33]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=20000
[00:05:33]               │ debg TestSubjects.exists(superDatePickerShowDatesButton)
[00:05:33]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerShowDatesButton"]') with timeout=2500
[00:05:33]               │ debg TestSubjects.click(superDatePickerShowDatesButton)
[00:05:33]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerShowDatesButton"]') with timeout=10000
[00:05:33]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerShowDatesButton"]') with timeout=10000
[00:05:33]               │ debg TestSubjects.exists(superDatePickerstartDatePopoverButton)
[00:05:33]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerstartDatePopoverButton"]') with timeout=2500
[00:05:33]               │ debg TestSubjects.click(superDatePickerendDatePopoverButton)
[00:05:33]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerendDatePopoverButton"]') with timeout=10000
[00:05:33]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerendDatePopoverButton"]') with timeout=10000
[00:05:33]               │ debg Find.findByCssSelector('div.euiPopover__panel-isOpen') with timeout=10000
[00:05:33]               │ debg TestSubjects.click(superDatePickerAbsoluteTab)
[00:05:33]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:05:33]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:05:33]               │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:05:33]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:33]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:33]               │ debg TestSubjects.setValue(superDatePickerAbsoluteDateInput, Oct 5, 2017 @ 20:35:12.176)
[00:05:33]               │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:05:33]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:33]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:34]               │ debg TestSubjects.click(superDatePickerstartDatePopoverButton)
[00:05:34]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerstartDatePopoverButton"]') with timeout=10000
[00:05:34]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerstartDatePopoverButton"]') with timeout=10000
[00:05:34]               │ debg Find.waitForElementStale with timeout=10000
[00:05:34]               │ debg Find.findByCssSelector('div.euiPopover__panel-isOpen') with timeout=10000
[00:05:34]               │ debg TestSubjects.click(superDatePickerAbsoluteTab)
[00:05:34]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:05:34]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:05:34]               │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:05:34]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:34]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:34]               │ debg TestSubjects.setValue(superDatePickerAbsoluteDateInput, Oct 5, 2017 @ 20:31:48.354)
[00:05:34]               │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:05:34]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:34]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:35]               │ debg TestSubjects.exists(superDatePickerApplyTimeButton)
[00:05:35]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=2500
[00:05:37]               │ debg --- retry.tryForTime error: [data-test-subj="superDatePickerApplyTimeButton"] is not displayed
[00:05:38]               │ debg TestSubjects.click(querySubmitButton)
[00:05:38]               │ debg Find.clickByCssSelector('[data-test-subj="querySubmitButton"]') with timeout=10000
[00:05:38]               │ debg Find.findByCssSelector('[data-test-subj="querySubmitButton"]') with timeout=10000
[00:05:38]               │ debg Find.waitForElementStale with timeout=10000
[00:05:38]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:38]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:38]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:38]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:05:38]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:05:38]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:38]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:38]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:38]               │ debg TestSubjects.click(clusterItemContainerElasticsearch > esOverview)
[00:05:38]               │ debg Find.clickByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:05:38]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:05:48]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:05:48]               │      Wait timed out after 10028ms
[00:05:49]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:05:59]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:05:59]               │      Wait timed out after 10011ms
[00:05:59]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:06:09]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:06:09]               │      Wait timed out after 10045ms
[00:06:10]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:06:20]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:06:20]               │      Wait timed out after 10044ms
[00:06:20]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:06:30]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:06:30]               │      Wait timed out after 10024ms
[00:06:31]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:06:41]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:06:41]               │      Wait timed out after 10014ms
[00:06:41]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:06:51]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:06:51]               │      Wait timed out after 10008ms
[00:06:52]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:07:02]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:07:02]               │      Wait timed out after 10006ms
[00:07:02]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:07:12]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:07:12]               │      Wait timed out after 10028ms
[00:07:13]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:07:23]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:07:23]               │      Wait timed out after 10006ms
[00:07:23]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:07:33]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:07:33]               │      Wait timed out after 10016ms
[00:07:34]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:07:44]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:07:44]               │      Wait timed out after 10060ms
[00:07:44]               │ info Taking screenshot "/dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/screenshots/failure/Monitoring app Elasticsearch overview mb _before all_ hook for _should have an Elasticsearch Cluster Summary Status with correct info_.png"
[00:07:45]               │ info Current URL is: http://localhost:61131/app/monitoring#/no-data?_g=(filters:!(),refreshInterval:(pause:!t,value:10000),time:(from:%272017-10-05T20:31:48.354Z%27,to:%272017-10-05T20:35:12.176Z%27))
[00:07:45]               │ info Saving page source to: /dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/failure_debug/html/Monitoring app Elasticsearch overview mb _before all_ hook for _should have an Elasticsearch Cluster Summary Status with correct info_.html
[00:07:45]               └- ✖ fail: Monitoring app Elasticsearch overview mb "before all" hook for "should have an Elasticsearch Cluster Summary Status with correct info"
[00:07:45]               │      Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:07:45]               │ Wait timed out after 10060ms
[00:07:45]               │     at /dev/shm/workspace/parallel/13/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
[00:07:45]               │     at runMicrotasks (<anonymous>)
[00:07:45]               │     at processTicksAndRejections (internal/process/task_queues.js:93:5)
[00:07:45]               │       at onFailure (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:17:9)
[00:07:45]               │       at retryForSuccess (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:57:13)
[00:07:45]               │       at Retry.try (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry.ts:32:14)
[00:07:45]               │       at Proxy.clickByCssSelector (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/find.ts:420:7)
[00:07:45]               │       at TestSubjects.click (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/test_subjects.ts:109:7)
[00:07:45]               │       at Context.<anonymous> (test/functional/apps/monitoring/elasticsearch/overview_mb.js:26:7)
[00:07:45]               │       at Object.apply (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)
[00:07:45]               │ 
[00:07:45]               │ 

Stack Trace

Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
Wait timed out after 10060ms
    at /dev/shm/workspace/parallel/13/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at onFailure (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:17:9)
    at retryForSuccess (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:57:13)
    at Retry.try (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry.ts:32:14)
    at Proxy.clickByCssSelector (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/find.ts:420:7)
    at TestSubjects.click (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/test_subjects.ts:109:7)
    at Context.<anonymous> (test/functional/apps/monitoring/elasticsearch/overview_mb.js:26:7)
    at Object.apply (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)

---

and 1 more failures, only showing the first 3.

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
monitoring	725.7KB	728.0KB	+2.4KB

History

• 💔 Build #117085 failed 80e946a
• 💔 Build #117046 failed 70d2638
• 💔 Build #115905 failed 8a6cf95
• 💔 Build #115810 failed 6067420

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream