Dashboard files in .ndjson format for ELK 8.x

[skmessage]

Is there a latest version of the dashboard JSON files in .ndjson format that is required for ELK 8.x?

[ActionScripted]

Not at all but happy to accept any changes via PR! I haven't worked on this in a long while.

[skmessage]

I converted the .json files to .ndjson. When trying to import any of the files in Management -> Saved Objects, I get the following error:

kibana_1         | [2023-10-11T08:28:14.618+00:00][WARN ][process] TypeError: Cannot destructure property 'title' of 'obj.attributes' as it is undefined.
kibana_1         |     at /usr/share/kibana/node_modules/@kbn/core-saved-objects-import-export-server-internal/src/import/lib/collect_saved_objects.js:38:7
kibana_1         |     at Transform.transform [as _transform] (/usr/share/kibana/node_modules/@kbn/utils/src/streams/filter_stream.js:20:33)
kibana_1         |     at Transform._write (node:internal/streams/transform:175:8)
kibana_1         |     at writeOrBuffer (node:internal/streams/writable:392:12)
kibana_1         |     at _write (node:internal/streams/writable:333:10)
kibana_1         |     at Transform.Writable.write (node:internal/streams/writable:337:10)
kibana_1         |     at Transform.ondata (node:internal/streams/readable:766:22)
kibana_1         |     at Transform.emit (node:events:514:28)
kibana_1         |     at addChunk (node:internal/streams/readable:324:12)
kibana_1         |     at readableAddChunk (node:internal/streams/readable:297:9)
kibana_1         |     at Transform.Readable.push (node:internal/streams/readable:234:10)
kibana_1         |     at node:internal/streams/transform:182:12
kibana_1         |     at Transform.transform [as _transform] (/usr/share/kibana/node_modules/@kbn/core-saved-objects-import-export-server-internal/src/import/lib/create_limit_stream.js:26:7)
kibana_1         |     at Transform._write (node:internal/streams/transform:175:8)
kibana_1         |     at writeOrBuffer (node:internal/streams/writable:392:12)
kibana_1         |     at _write (node:internal/streams/writable:333:10)
kibana_1         |     at Transform.Writable.write (node:internal/streams/writable:337:10)
kibana_1         |     at Readable.ondata (node:internal/streams/readable:766:22)
kibana_1         |     at Readable.emit (node:events:514:28)
kibana_1         |     at Readable.read (node:internal/streams/readable:539:10)
kibana_1         |     at flow (node:internal/streams/readable:1023:34)
kibana_1         |     at resume_ (node:internal/streams/readable:1004:3)
kibana_1         |     at processTicksAndRejections (node:internal/process/task_queues:82:21)

I am using release-7.x branch from https://github.com/deviantony/docker-elk/tree/release-7.x. Any suggestions on how to fix the same? The ui-dashboards.ndjson file (cannot attach due to unsupported file type)

{"_id":"b6b2aea0-788f-11e8-a41e-1930b01c3fee","_type":"dashboard","_source.title":"[Filebeat Postfix] Overview","_source.hits":0,"_source.description":"","_source.panelsJSON":"[{\"panelIndex\":\"4\",\"gridData\":{\"x\":0,\"y\":8,\"w\":48,\"h\":11,\"i\":\"4\"},\"embeddableConfig\":{},\"id\":\"0609c7b0-7893-11e8-a41e-1930b01c3fee\",\"title\":\"Status over time\",\"type\":\"visualization\",\"version\":\"6.3.0\"},{\"panelIndex\":\"7\",\"gridData\":{\"x\":0,\"y\":0,\"w\":10,\"h\":8,\"i\":\"7\"},\"embeddableConfig\":{\"spy\":null},\"id\":\"2261e870-78ac-11e8-a41e-1930b01c3fee\",\"title\":\"Messages total\",\"type\":\"visualization\",\"version\":\"6.3.0\"},{\"panelIndex\":\"8\",\"gridData\":{\"x\":10,\"y\":0,\"w\":30,\"h\":8,\"i\":\"8\"},\"embeddableConfig\":{},\"id\":\"618e79d0-78ae-11e8-a41e-1930b01c3fee\",\"title\":\"Message status totals (click to filter)\",\"type\":\"visualization\",\"version\":\"6.3.0\"},{\"panelIndex\":\"9\",\"gridData\":{\"x\":0,\"y\":31,\"w\":48,\"h\":17,\"i\":\"9\"},\"embeddableConfig\":{},\"id\":\"4cdea1d0-7891-11e8-a41e-1930b01c3fee\",\"title\":\"Logs\",\"type\":\"search\",\"version\":\"6.3.0\"},{\"panelIndex\":\"10\",\"gridData\":{\"x\":40,\"y\":0,\"w\":8,\"h\":8,\"i\":\"10\"},\"title\":\"Manual (click to filter)\",\"embeddableConfig\":{},\"id\":\"480e8b10-78b0-11e8-a41e-1930b01c3fee\",\"type\":\"visualization\",\"version\":\"6.3.0\"},{\"panelIndex\":\"11\",\"gridData\":{\"x\":24,\"y\":19,\"w\":24,\"h\":12,\"i\":\"11\"},\"embeddableConfig\":{},\"id\":\"49274ce0-7896-11e8-a41e-1930b01c3fee\",\"title\":\"Top senders\",\"type\":\"visualization\",\"version\":\"6.3.0\"},{\"panelIndex\":\"12\",\"gridData\":{\"x\":0,\"y\":19,\"w\":24,\"h\":12,\"i\":\"12\"},\"embeddableConfig\":{},\"id\":\"cf5cb2f0-7891-11e8-a41e-1930b01c3fee\",\"title\":\"Hosts\",\"type\":\"visualization\",\"version\":\"6.3.0\"}]","_source.optionsJSON":"{\"darkTheme\":false,\"hidePanelTitles\":false,\"useMargins\":true}","_source.version":1,"_source.timeRestore":false,"_source.kibanaSavedObjectMeta.searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[],\"highlightAll\":true,\"version\":true}"}

[ActionScripted]

I don't have a running version to test with, but I think we need to adjust the format/content of the ndjson to match the newer Kibana structure:

{
  "attributes": {
    "title": "[Filebeat Postfix] Overview",
    "hits": 0,
    "description": "",
    "panelsJSON": "…",
    "optionsJSON": "…",
    "version": 1,
    "timeRestore": false,
    "kibanaSavedObjectMeta": {
      "searchSourceJSON": "…"
    }
  },
  "id": "b6b2aea0-788f-11e8-a41e-1930b01c3fee",
  "type": "dashboard"
}

[ActionScripted]

Really appreciate you working to get things fixed up, btw.

[skmessage]

With release-7.x branch, I get:

Sorry, there was an error

The file could not be processed due to error: "Bad Request: Unexpected end of JSON input"

Is there a way to validate the .ndjson file before trying to import the same?

[ActionScripted]

If you've got a shell with Python handy you could do something like:

import json
from pathlib import Path

def validate_ndjson(file_path):
    try:
        # Reading and parsing each line as JSON
        [json.loads(line) for line in Path(file_path).read_text().splitlines()]
        return "Valid NDJSON"
    except json.JSONDecodeError:
        return "Invalid NDJSON"

# Validating the NDJSON file
validate_ndjson("foo.ndjson")

[skmessage]

I have created a PR that cleanly imports api-combined.ndjson, ui-dashboards.ndjson, and ui-visualizations.ndjson files in Elasticsearch v 8.10.2.

#16

But, when I try to open Dashboards -> [Filebeat Postfix] Overview it throws the following error:

Cannot read properties of undefined (reading 'searchSourceJSON')

I removed the searchSourceJSON field from kibanaSavedObjectMeta key. Is this mandatory?