Missing needed index fields.

[rong0312]

Problem description

Missing needed fields, resulting with dashboard errors and efficient logging.

Extra information

Comparings docker swarm ELK vs normal hosted ELK.

Client side:
I am Using the same host & metricbeat configuration to send data to deviantony/docker-elk docker stack (missing fields) and to another plain vanilla ELK which is host based for comparison.
Compose file is attached bellow,

In this issue i will refer to "Metricbeat" for mongodb but other beats didn't
fulfil other dashboards as well.

Stack configuration

docker-compose.yml:

version: '3.7'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.5.4
    ports:
      - "9200:9200"
      - "9300:9300"
    volumes:
      - ./elasticsearch/data:/usr/share/elasticsearch/data
    environment:
      ES_JAVA_OPTS: "-Xmx1500m -Xms1500m"
      cluster.name: "elk-cluster"
      network.host: 0.0.0.0
      discovery.zen.ping.unicast.hosts: tasks.elasticsearch
      discovery.type: zen
      discovery.zen.minimum_master_nodes: 2
      xpack.monitoring.enabled: "true"
      xpack.monitoring.collection.enabled: "true"
    networks:
      - elk
    deploy:
      mode: global
      restart_policy:
        condition: on-failure

  logstash:
    image: docker.elastic.co/logstash/logstash:6.5.4
    ports:
      - "5000:5000"
      - "5044:5044"
      - "9600:9600"
    configs:
      - source: logstash_pipeline
        target: /usr/share/logstash/pipeline/logstash.conf
    environment:
      LS_JAVA_OPTS: "-Xmx500m -Xms500m"
      http.host: "0.0.0.0"
      xpack.monitoring.elasticsearch.url: http://elasticsearch:9200
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: on-failure

  kibana:
    image: docker.elastic.co/kibana/kibana:6.5.4
    ports:
      - "5601:5601"
    environment:
      server.name: kibana
      server.host: "0"
      elasticsearch.url: http://elasticsearch:9200
      xpack.monitoring.ui.container.elasticsearch.enabled: "true"
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: on-failure

configs:
  logstash_pipeline:
    file: ./logstash/pipeline/logstash.conf

networks:
  elk:
    driver: overlay

------------------------END OF COMPOSE --------------------------

Additional information and pics:

Both cases came from same metricbeat configuration and host as mentioned.
Difference between docker stack and plain host ELK:

Docker stack fields:

Plain host fields:

Please notice the fields! any idea why there is such gap?
did anyone else faced issues related to uploaded dashboards from beats?
missing fields?

even when i query kibana: GET metricbeat-6.5.4-2019.01.17/_mapping
I see huge difference between the solutions (10x more lines in plain hosted ELK).

Any guidance will be welcome and appreciated.
Thanks alot!
Ron

Dashboard:

[antoineco]

Hi @rong0312
Metricbeat is not part of this stack. Would you mind asking this question in the Elastic community forums?

[rong0312]

Hey @antoineco ,
I dont believe its a beat problem, as i tried to explain i compared 2 environments with
same beat client but the results were different.
more over, other beats behave alike.

How are your beats display data? did you find any troubles?

[antoineco]

Got it. Do you see the same behaviour with a single ES node? (and without discovery.zen.minimum_master_nodes: 2)

[rong0312]

Sadly same results

By the way, seems like host monitoring not working for me aswell

Logs seems ok due.

[antoineco]

Any error in the metricsbeat shipper?

[rong0312]

Update;
All ok with metricbeat logs.
Just found that in logstash's log:

elk_logstash.1.r7azmc5woyoj@BE-SMTP1-DEV    | [2019-01-17T14:11:47,384][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"metricbeat-6.5.4-2019.01.17", :_type=>"doc", :routing=>nil}, #<LogStash::Event:0x510783f3>], :response=>{"index"=>{"_index"=>"metricbeat-6.5.4-2019.01.17", "_type"=>"doc", "_id"=>"EC8nXGgBDQ5BGs6VQ_c9", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [system.process.memory.rss.pct] cannot be changed from type [float] to [long]"}}}}

I have no idea what causes that on docker ELK, how ever I believe there is still no reason for difference between hosted and dockered ELK.

[antoineco]

I see a similar issue has already been raised at elastic/beats#5032. The second comment is particularly interesting:

Metricbeat comes with its own template, which you should make sure to load in ES. Normally, when sending the data directly to ES, this happens automatically, but not when using Logstash as an intermediary point.

I also found the following comment in the Elastic discussion board:

If you want to make sure that a field is mapped in a particular way you should use an index template. ES picks the data type of a field based on the first occurrence of the field. Without explicitly set mappings, ES's automapper will let the data type of the first document with a particular field set the precedent. In other words, it seems like system.process.cpu.total.pct sometimes contains an integer, and when the day's first document with that field happens to be an integer you'll run into trouble later on.

Unfortunately for you if the first document was a long all subsequent float documents will be ignored.

---

Like I said earlier, this repo does not include Metricbeat, nor can we guarantee that the default Logstash/Elasticsearch settings are 100% compatible with Metricbeat.

Please make sure you followed all the steps described in the official Getting Started guide. Especially "Step 3: Load the index template in Elasticsearch".

[rong0312]

Ok, thank you very much for those insights,
I will try different beats next week and i will post here the results.

Meanwhile, in case some metricbeat user sees this issue and knows somthing regards ES/logstash settings fixes i will be glad to here from him/her.

[rong0312]

I managed to deploy beat agents (winlogbeat and metricbeat) on a temporary windows server 2012.
After some configuration changes (beat-wise), my dashboards look allright and all fields seems to be present.

I will work on my docker-metricbeat to understand what is wrong on docker environment.

[rong0312]

Iv installed more beats as containers, all seems to work just fine,
I probably had a problem with some specific beats.
thank you alot for your time @antoineco , im closing this issue for now.