Unable to open any Kibana dashboard

[Rafael-Bardisa]

I cannot view any of the default T-pot dashboards in kibana. I can load the dashboards page just fine, but any other page does not load. I have no idea where to start looking for the problem. I tried running the update.sh script but the problem still persists.

My elastic search version is 8.6.0, and my kibana version is 8.6.2. Below are screenshots of my problem:

Kibana dashboard page:

Kibana T-pot dashboard, with errors:

Trying to access the canvas page:

The state of the top machine:

dps.sh:

glances:

Thanks in advance

[t3chn0m4g3]

What error do you receive when trying to load the T-Pot Dashboard? We do restrict the ELK stack to a bare minimum necessary to display the T-Pot events. In case you need all the ELK stack features (i.e. canvas) we recommend setting up your own instance.
You should check if you have all the latest docker images downloaded. If your Kibana and Elasticsearch are running different versions then somehow the latest version of ES has not been downloaded yet. Please follow the Readme how to update.

[Rafael-Bardisa]

I've rerun the update.sh script, and it seemed to execute without problems

When I enter the dashboards page, I get the following, which I assume is normal:

However, when I try to load the T-pot dashboard, the error is that some scripts are not found, and nothing shows up in the browser:

I've run docker logs -f for the kibana container and I get the following:

[admin@crowdeddrug:~]$ sudo docker logs -f kibana
sudo: unable to resolve host crowdeddrug: Name or service not known
[2023-06-05T08:47:33.807+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
[2023-06-05T08:48:06.062+00:00][INFO ][plugins-service] Plugin "cloudChat" is disabled.
[2023-06-05T08:48:06.062+00:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled.
[2023-06-05T08:48:06.062+00:00][INFO ][plugins-service] Plugin "cloudFullStory" is disabled.
[2023-06-05T08:48:06.063+00:00][INFO ][plugins-service] Plugin "cloudGainsight" is disabled.
[2023-06-05T08:48:06.126+00:00][INFO ][plugins-service] Plugin "profiling" is disabled.
[2023-06-05T08:48:06.307+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
[2023-06-05T08:48:06.492+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2023-06-05T08:48:06.574+00:00][WARN ][config.deprecation] You should set server.basePath along with server.rewriteBasePath. Starting in 7.0, Kibana will expect that all requests start with server.basePath rather than expecting you to rewrite the requests in your reverse proxy. Set server.rewriteBasePath to false to preserve the current behavior and silence this warning.
[2023-06-05T08:48:06.577+00:00][WARN ][config.deprecation] Setting "kibana.autocompleteTerminateAfter" has been replaced by "unifiedSearch.autocomplete.valueSuggestions.terminateAfter"
[2023-06-05T08:48:06.578+00:00][WARN ][config.deprecation] Setting "kibana.autocompleteTimeout" has been replaced by "unifiedSearch.autocomplete.valueSuggestions.timeout"
[2023-06-05T08:48:07.253+00:00][INFO ][plugins-system.standard] Setting up [127] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,fieldFormats,expressions,screenshotting,dataViews,embeddable,uiActionsEnhanced,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,eventAnnotation,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,discover,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,dataVisualizer,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,osquery,observability,aiops,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]
[2023-06-05T08:48:07.280+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: b98cdd49-b70a-4c56-9df5-a53dd5ce3cb6
[2023-06-05T08:48:07.426+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-06-05T08:48:07.427+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2023-06-05T08:48:07.556+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-06-05T08:48:07.557+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2023-06-05T08:48:07.575+00:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-06-05T08:48:07.617+00:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-06-05T08:48:07.627+00:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
[2023-06-05T08:48:07.856+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-06-05T08:48:07.859+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
[2023-06-05T08:48:07.867+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-06-05T08:48:07.947+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2023-06-05T08:48:08.042+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
[2023-06-05T08:48:09.681+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 22.04 OS. Automatically enabling Chromium sandbox.
[2023-06-05T08:48:10.237+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2023-06-05T08:48:10.239+00:00][INFO ][savedobjects-service] Starting saved objects migrations
[2023-06-05T08:48:10.380+00:00][INFO ][savedobjects-service] [.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 75ms.
[2023-06-05T08:48:10.418+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 107ms.
[2023-06-05T08:48:10.484+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 104ms.
[2023-06-05T08:48:10.616+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 198ms.
[2023-06-05T08:48:10.898+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 414ms.
[2023-06-05T08:48:10.909+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 293ms.
[2023-06-05T08:48:10.918+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 20ms.
[2023-06-05T08:48:10.923+00:00][INFO ][savedobjects-service] [.kibana] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 5ms.
[2023-06-05T08:48:10.929+00:00][INFO ][savedobjects-service] [.kibana] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 6ms.
[2023-06-05T08:48:10.930+00:00][INFO ][savedobjects-service] [.kibana] Migration completed after 625ms
[2023-06-05T08:48:10.934+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 25ms.
[2023-06-05T08:48:10.935+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 1ms.
[2023-06-05T08:48:10.938+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 3ms.
[2023-06-05T08:48:10.939+00:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 628ms
[2023-06-05T08:48:10.955+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
[2023-06-05T08:48:10.958+00:00][INFO ][plugins-system.standard] Starting [127] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,fieldFormats,expressions,screenshotting,dataViews,embeddable,uiActionsEnhanced,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,eventAnnotation,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,discover,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,dataVisualizer,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,osquery,observability,aiops,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]
[2023-06-05T08:48:15.533+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Sender-1.1.0 scheduled with interval 1h
[2023-06-05T08:48:15.622+00:00][INFO ][plugins.monitoring.monitoring] config sourced from: production cluster
[2023-06-05T08:48:18.891+00:00][INFO ][http.server.Kibana] http server running at http://0.0.0.0:5601
[2023-06-05T08:48:18.949+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Logger-Task scheduled with interval 15m
[2023-06-05T08:48:18.982+00:00][INFO ][status] Kibana is now degraded
[2023-06-05T08:48:19.041+00:00][ERROR][plugins.ml] Error Initializing ML saved objects {"name":"ResponseError","meta":{"body":{"error":"Incorrect HTTP method for uri [/_ml/anomaly_detectors] and method [GET], allowed: [POST]","status":405},"statusCode":405,"headers":{"x-opaque-id":"unknownId","allow":"POST","x-elastic-product":"Elasticsearch","content-type":"application/json","content-length":"113"},"meta":{"context":null,"request":{"params":{"method":"GET","path":"/_ml/anomaly_detectors","querystring":"","headers":{"user-agent":"Kibana/8.6.2","x-elastic-product-origin":"kibana","x-elastic-client-meta":"es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1","x-opaque-id":"unknownId","accept":"application/vnd.elasticsearch+json; compatible-with=8,text/plain"}},"options":{"opaqueId":"unknownId","headers":{"x-elastic-product-origin":"kibana","user-agent":"Kibana/8.6.2","x-elastic-client-meta":"es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1"}},"id":7},"name":"elasticsearch-js","connection":{"url":"http://elasticsearch:9200/","id":"http://elasticsearch:9200/","headers":{},"status":"alive"},"attempts":0,"aborted":false},"warnings":null}}
[2023-06-05T08:48:19.509+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Starting monitoring stats collection
[2023-06-05T08:48:19.622+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
[2023-06-05T08:48:19.804+00:00][INFO ][plugins.ruleRegistry] Installed common resources shared between all indices
[2023-06-05T08:48:19.805+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.uptime.alerts
[2023-06-05T08:48:19.806+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-security.alerts
[2023-06-05T08:48:19.807+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .preview.alerts-security.alerts
[2023-06-05T08:48:19.807+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.logs.alerts
[2023-06-05T08:48:19.807+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.metrics.alerts
[2023-06-05T08:48:19.808+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.apm.alerts
[2023-06-05T08:48:19.918+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.logs.alerts
[2023-06-05T08:48:19.919+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.apm.alerts
[2023-06-05T08:48:19.933+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-security.alerts
[2023-06-05T08:48:19.948+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.uptime.alerts
[2023-06-05T08:48:19.949+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.metrics.alerts
[2023-06-05T08:48:20.034+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .preview.alerts-security.alerts
[2023-06-05T08:48:21.195+00:00][INFO ][plugins.ml] Task ML:saved-objects-sync-task: scheduled with interval 1h
[2023-06-05T08:48:22.679+00:00][ERROR][plugins.taskManager] Task ML:saved-objects-sync "ML:saved-objects-sync-task" failed: ResponseError: {"error":"Incorrect HTTP method for uri [/_ml/anomaly_detectors] and method [GET], allowed: [POST]","status":405}
[2023-06-05T08:48:23.539+00:00][ERROR][plugins.taskManager] Task apm-telemetry-task "apm-telemetry-task" failed: TypeError: Cannot read properties of undefined (reading 'logger')
[2023-06-05T08:48:25.016+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-06-05T08:51:22.689+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"total_all_statuses":0,"num_host_urls":0}}

I'm not sure how to interpret the above output, but one error line suggests kibana is using elasticsearch 8.4.0. I don't know if this could be related to kibana objects being outdated, but I do not have access to the stack management window either. Is there a way to force the elasticsearch image to upgrade?

[t3chn0m4g3]

I cannot reproduce this. Fresh install runs as fine as any other updated installation I am running.

[Rafael-Bardisa]

thank you for looking into it, will try to snapshot the data and load it on a fresh T-pot instance