Importing dashboards from icingabeat renders kibana 6.0 unusable

[jpcarey]

Kibana version:
6.0.0

Elasticsearch version:
6.0.0 (upgraded from 5.6.4)

Description of the problem including expected versus actual behavior:

Steps to reproduce:

1. Run fresh elasticsearch + kibana 5.6.4, and use upgrade assistant to reindex system indices
2. Stop kibana/elasticsearch, and copy the elasticsearch data folder to fresh 6.0.0 install
3. Startup working 6.0.0 system (setup passwords, etc)
4. Download latest icingabeat and dashboard zip and extract both (https://github.com/Icinga/icingabeat/releases/tag/v1.1.1)
5. Use ./import_dashboards -es 'http://localhost:9200' -user elastic -pass changeme -dir ../../icingabeat-dashboards-1.1.1 (which will fail)

Import directory ../../icingabeat-dashboards-1.1.1/index-pattern
Import index to /.kibana/index-pattern/icingabeat-* from ../../icingabeat-dashboards-1.1.1/index-pattern/icingabeat.json

Error importing directory ../../icingabeat-dashboards-1.1.1: Failed to import index-pattern: Failed to load directory ../../icingabeat-dashboards-1.1.1/index-pattern:
error loading ../../icingabeat-dashboards-1.1.1/index-pattern/icingabeat.json: couldn't load json. Error: 404 Not Found
Exiting

6. Kibana will report "Your Kibana index is out of date, reset it or use the X-Pack upgrade assistant.". This cannot be fixed without dumping all the .kibana data, deleting the index, and reindex all the data.
   

No dashboards or data is actually added to the .kibana index, but the mapping gets modified.

This will reproduce the issue. The index must be upgraded from 5.x in order for this to happen.

PUT .kibana/_mapping/search
{
  "_all": {
    "enabled": true
  },
  "properties": {
    "hits": {
      "type": "integer"
    },
    "version": {
      "type": "integer"
    }
  }
}

If the .kibana index is created in the 6.x, elasticsearch rejects it (which avoids putting Kibana into an unusable state):

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Enabling [_all] is disabled in 6.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field."
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Enabling [_all] is disabled in 6.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field."
  },
  "status": 400
}

[tylersmalley]

Where does the import_dashboards script come from? In 6.0 we no longer use types as indices not be created with them in 6.0. As of beats 6.0, they use the Dashboard Import API and do not directly interact with the Kibana index.

[tylersmalley]

Also, starting in 5.6.5 we are setting index.mapping.single_type: true when migrating the Kibana index through the Upgrade Assistant to prevent the accidental addition of types.

[jpcarey]

@tylersmalley https://github.com/Icinga/icingabeat/releases/tag/v1.1.1. Beats used to use an ./import_dashboard script, which is likely to be widely used with community beats. The index.mapping.single_type: true addition should help, but all users that upgraded in < 5.6.5 will still have this risk.

[tylersmalley]

@jpcarey, the beats team moved to using the import/export Dashboard API's which were created for their use case.

@tsg, how would a beat go about switching to using this new API? Are there helpers, an updated import_dashboard script, or would they just need to call the API manually?

[tylersmalley]

Closing, the plugin should use the import/export dashboard API. I suspect there are helpers in libbeat for this.