Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include ip and boolean type when generating index pattern #10995

Merged
merged 6 commits into from
Mar 5, 2019

Conversation

cwurm
Copy link
Contributor

@cwurm cwurm commented Feb 28, 2019

A number of fields in our fields.yml files now are type: ip (e.g. source.ip) and type: boolean, but the code generating index patterns does not know about these types yet and so does not add a type to the index pattern for those fields at all. This leads to errors in Kibana when looking at dashboards that contain references to those fields.

Output of ./auditbeat export index-pattern:

Before:

  {
    "aggregatable": true,
    "analyzed": false,
    "count": 0,
    "doc_values": true,
    "indexed": true,
    "name": "source.ip",
    "scripted": false,
    "searchable": true
  },

After:

  {
    "aggregatable": true,
    "analyzed": false,
    "count": 0,
    "doc_values": true,
    "indexed": true,
    "name": "source.ip",
    "scripted": false,
    "searchable": true,
    "type": "ip"
  },

@cwurm cwurm added bug review libbeat needs_backport PR is waiting to be backported to other branches. labels Feb 28, 2019
@cwurm cwurm requested a review from a team as a code owner February 28, 2019 16:02
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. I just thought is was a quirk of Kibana where data needed to be indexed first.

@webmat
Copy link
Contributor

webmat commented Mar 1, 2019

Tests need updating, just adjusting the expected value to include IP: https://beats-ci.elastic.co/job/elastic+beats+pull-request+multijob-linux/5477/beat=libbeat,label=linux-immutable/console

But thanks for looking into this, I also thought this was a quirk :-)

@cwurm
Copy link
Contributor Author

cwurm commented Mar 1, 2019

On closer inspection, I noticed the boolean type was missing as well, so I added it. Every field in the Auditbeat index pattern now has a type.

I checked Metricbeat as well and some fields there are still missing their type. At least one cause is when type: object and object_type: keyword. But I'm not going to investigate that one further at the moment. /cc @ruflin in case you want to take a look

@cwurm cwurm changed the title Include type: ip when generating index pattern Include ip and boolean type when generating index pattern Mar 1, 2019
Copy link
Contributor

@webmat webmat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks, this is a great find :-)

@cwurm cwurm force-pushed the index_pattern_type_ip branch 2 times, most recently from 9f04f61 to 35698a0 Compare March 4, 2019 12:57
@cwurm cwurm force-pushed the index_pattern_type_ip branch from 35698a0 to fb2f1bb Compare March 5, 2019 12:59
@cwurm cwurm merged commit e98026e into elastic:master Mar 5, 2019
@cwurm cwurm deleted the index_pattern_type_ip branch March 5, 2019 14:56
@cwurm cwurm added v7.2.0 and removed needs_backport PR is waiting to be backported to other branches. labels Mar 5, 2019
cwurm pushed a commit to cwurm/beats that referenced this pull request Mar 5, 2019
)

A number of fields in our `fields.yml` files now are `type: ip` (e.g. `source.ip`) and `type: boolean`, but the code generating index patterns does not know about these types yet and so does not add a `type` to the index pattern for those fields at all. This leads to errors in Kibana when looking at dashboards that contain references to those fields.

(cherry picked from commit e98026e)
@cwurm cwurm added the v7.0.0 label Mar 5, 2019
@cwurm cwurm added the v6.7.0 label Mar 5, 2019
cwurm pushed a commit to cwurm/beats that referenced this pull request Mar 6, 2019
)

A number of fields in our `fields.yml` files now are `type: ip` (e.g. `source.ip`) and `type: boolean`, but the code generating index patterns does not know about these types yet and so does not add a `type` to the index pattern for those fields at all. This leads to errors in Kibana when looking at dashboards that contain references to those fields.

(cherry picked from commit e98026e)
cwurm pushed a commit to cwurm/beats that referenced this pull request Mar 6, 2019
)

A number of fields in our `fields.yml` files now are `type: ip` (e.g. `source.ip`) and `type: boolean`, but the code generating index patterns does not know about these types yet and so does not add a `type` to the index pattern for those fields at all. This leads to errors in Kibana when looking at dashboards that contain references to those fields.

(cherry picked from commit e98026e)
cwurm pushed a commit that referenced this pull request Mar 6, 2019
…g index pattern (#11089)

Cherry-pick of PR #10995 to 7.x branch. Original message: 

A number of fields in our `fields.yml` files now are `type: ip` (e.g. `source.ip`) and `type: boolean`, but the code generating index patterns does not know about these types yet and so does not add a `type` to the index pattern for those fields at all. This leads to errors in Kibana when looking at dashboards that contain references to those fields.
cwurm pushed a commit that referenced this pull request Mar 6, 2019
…11090)

A number of fields in our `fields.yml` files now are `type: ip` (e.g. `source.ip`) and `type: boolean`, but the code generating index patterns does not know about these types yet and so does not add a `type` to the index pattern for those fields at all. This leads to errors in Kibana when looking at dashboards that contain references to those fields.

(cherry picked from commit e98026e)
cwurm pushed a commit that referenced this pull request Mar 7, 2019
…g index pattern (#11091)

Cherry-pick of PR #10995 to 6.7 branch. Original message: 

A number of fields in our `fields.yml` files now are `type: ip` (e.g. `source.ip`) and `type: boolean`, but the code generating index patterns does not know about these types yet and so does not add a `type` to the index pattern for those fields at all. This leads to errors in Kibana when looking at dashboards that contain references to those fields.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants