Improve Kibana deployment docs

[tbragin]

Right now we only talk about HA between Kibana and multiple ES nodes using the client node on the Kibana side. We should expand the docs to talk about the following scenarios:

• Upgrade documentation, especially in the context of rolling upgrades
• Best practices, such as ensuring NTP is running on the Kibana server
• Hardware sizing guidance
• Setting up more than one Kibana instance pointing to the same cluster and using the same .kibana index for Kibana instance HA (Add a new documentation page on Load Balancing Across Multiple Kibana Instances #9129)
• Using HA proxy or LB instead of the client nodes for load balancing
• How to architect Kibana+ES to use cross cluster search to unify reading across multiple ES clusters

[nellicus]

great to see this coming @tbragin

can we add the supported OS matrix to the items list?

Kibana server is the only product missing here

[CVTJNII]

Allowing writes from Kibana instances when using tribe nodes should also be addressed, please, as the bug trail I followed (#3114) ended here. The .kibana index needs to be writable to allow features like saving dashboards. I'm currently solving this by intercepting the .kibana endpoint with a proxy and sending that directly to a cluster, bypassing the tribe node.

[LucaWintergerst]

This sounds very helpful. Especially the HA stuff!

@CVTJNII So even when I

1. create the kibana index by pointing the kibana to one single cluster first
2. create 1 search, 1 visualization, and 1 dashboard
3. point the kibana to the tribe node
   saving further dashboards will not work?

[CVTJNII]

@LucaWintergerst That is correct. However, if you set up a proxy to redirect the Kibana index to a dedicated "write" cluster then not only will saving dashboards work but no special steps around creating the index will be required, either. I did this by intercepting /.kibana with HAProxy. One key thing to note is to set the tribe nodes as a backup in your kibana backend config so that Kibana doesn't flap if something goes wrong with the write clients due to being able to read some of it's index data through the tribe nodes.

[LucaWintergerst]

@CVTJNII I managed to partially solve this problem by using puppet. Everytime I add a new instance, an elasticsearch index with the specific mapping is created and a docker container with kibana started. The only thing we are still struggling with is the shorten URL feature.
But since we do already have a HAProxy in front of everything, we will just implement the rule you mentioned. Looks like the easier way. Thanks for your help! And sorry for being a little off topic here

[pierro42100]

@CVTJNII

One key thing to note is to set the tribe nodes as a backup in your kibana backend config so that Kibana doesn't flap if something goes wrong with the write clients due to being able to read some of it's index data through the tribe nodes.

I created the .kibana index by connecting it to a single cluster A (like @LucaWintergerst said). The .kibana isn't created in the tribe node so if the cluster A get down, the Kibana doesn't work anymore.
So, is it possible to create this index in the tribe node ? Or, what is the Kibana conf to set the tribe node as a backup ?

[LucaWintergerst]

I think your best bet at the moment would be to:

• redirect all kibana requests to Cluster A (read only)
• to tribe endpoint if Cluster A is down
• snapshot .kibana index in Cluster A every N Minutes (or once per day)
• load snapshot in Cluster B every N Minutes

Result: If Cluster A goes down, you can still load kibana boards from cluster B until A is fixed again. I would discourage you from giving write access to cluster Bs Kibana index, as you'd have to snapshot the index the other way around then..So during your downtime you cant create new boards.

Let me know if there are any better solutions. Keep in mind that this is a more simple solution. I can think of something better that would be a whole lot more complicated to set up.

[CVTJNII]

So what I'm currently doing is:

• Redirect all .kibana index requests to Cluster A read/write
• Redirect .kibana to the Tribe endpoint if the Cluster A client nodes are failing the healthcheck. This avoids Kibana flapping if the A cluster clients are down.

Redirecting both reads and writes to the A cluster client nodes allows Kibana to set up it's index normally, no special handling is required with this setup. We currently have the snapshot method @LucaWintergerst mentioned planned via curator but not set up yet.

If you're doing this with HAProxy it will look something like this:

 # This HAProxy config sets up a proxy intended to sit between Kibana and a Tribe node
# to allow .kibana to be writable for saved dashboards and settings and such.

global
  chroot /var/lib/haproxy
  user   haproxy
  group  haproxy

defaults
  timeout connect  2s
  timeout server  15s
  timeout server  15s
  timeout client  15s

frontend fe_es
  bind 0.0.0.0:9200
  mode http

  # ACL for the Kibana index
  acl kibana path_beg /.kibana

  use_backend kibana_clients if kibana
  default_backend tribe

# Backend for the .kibana index, this talks directly to client nodes in the write cluster.
backend kibana_clients
  mode http
  option httpchk GET /_cluster/health
  http-check expect status 200

  balance roundrobin
  server client0 es-client1.domain:9200 check inter 1s
  server client1 es-client2.domain:9200 check inter 1s
  server client2 es-client3.domain:9200 check inter 1s
  # Include the tribe nodes as a backups.
  # In the event the clients go down this will allow Kibana to read the .kibana index through the tribe node
  # WARNING: If this is not present, the clients are down, but the .kibana index is available through the
  #  tribe node then Kibana 4 will flap from red to green back to red endlessly.
  server tribe0 es-tribe1.domain:9200 check inter 1s backup
  server tribe1 es-tribe2.domain:9200 check inter 1s backup
  server tribe2 es-tribe3.domain:9200 check inter 1s backup

# Main tribe backend for all non-.kibana queries
backend tribe
  mode http
  option httpchk GET /_cluster/health
  http-check expect status 200

  server tribe0 es-tribe1.domain:9200 check inter 1s
  server tribe1 es-tribe2.domain:9200 check inter 1s
  server tribe2 es-tribe3.domain:9200 check inter 1s

This is my config where es-client.domain is the A cluster client nodes (the hostnames are sanitized) and the tribe nodes are configured (configuration not shown) for multiple clusters.

[tbragin]

For Kibana minor version upgrades, here is the rough procedure that should work and result in no Kibana downtime for the manual installation of Kibana:

1. Upgrade Elasticsearch first, if your new version of Kibana requires a new version of Elasticsearch
2. Backup Kibana
   • zip and back up kibana folder
   • snapshot .kibana index
3. Download new bits
4. Diff and merge over settings from old kibana.yml to kibana.yml
5. Re-install any plugins
6. Start new Kibana
7. Stop old Kibana (for minor versions, the previous version of Kibana should still work with the new version of ES, assuming ES did not introduce breaking changes, which it should not in minor versions).

Procedure will be slightly different for packages.

@LeeDr Could you double-check the steps and let me know if this works, for instance for 4.3.x to 4.5.x upgrades? We really should document this procedure in the Kibana docs, perhaps here?
https://www.elastic.co/guide/en/kibana/current/setup.html

Also, do you have steps for package-based upgrade process?

cc: @epixa

[ppf2]

6. Start new Kibana

Note that for step 6 above, if doing this on the same server (likely), the new Kibana will have to be configured to run on a different port than the existing instance to avoid conflicts. Which means that there needs to be some proxy in front to map to a different Kibana port from the past to ensure no downtime. If not, users will have to be instructed to access Kibana from a different port during the upgrade and then Step 8 will be to restart the new Kibana to have it run on the original port (i.e. downtime?)

[schersh]

Closing out this issue because it doesn't have any recent activity. If you still feel this issue needs to be addressed, feel free to open it back up.