kibana does not start due to wrong permissions of optimize dir

[meierfra-ergon]

Kibana version: 7.9.1

Elasticsearch version: 7.9.1

Server OS version: CentOS-7

Original install method (e.g. download page, yum, from source, etc.): yum

Describe the bug:
After installing kibana RPM on Centos 7 and starting it via systemctl start kibana, kibana does not start and logs the following error, seen via systemctl status kibana:

Sep 16 09:20:19 airlock-waf kibana[4548]: log   [07:20:19.277] [fatal][root] { [Error: EACCES: permission denied, unlink '/usr/share/kibana/optimize/bundles/timelion.style.css']
Sep 16 09:20:19 airlock-waf kibana[4548]: errno: -13,
Sep 16 09:20:19 airlock-waf kibana[4548]: code: 'EACCES',
Sep 16 09:20:19 airlock-waf kibana[4548]: syscall: 'unlink',
Sep 16 09:20:19 airlock-waf kibana[4548]: path: '/usr/share/kibana/optimize/bundles/timelion.style.css' }
Sep 16 09:20:19 airlock-waf kibana[4548]: FATAL  Error: EACCES: permission denied, unlink '/usr/share/kibana/optimize/bundles/timelion.style.css'

The problem is, that the directory /usr/share/kibana/optimize/ is owned by root. in previous versions as 7.8.1 this was owned by kibana. While comparing the post_install.sh scripts from 7.8 to 7.9 ([1],[2]) one sees, that in 7.8 there was a chown that is missing in 7.9 [3]. And the other chown for pluginsDir is also missing.

BTW I think this issue is similiar to #77392.

update: this does not happen with the default kibana config (/etc/kibana/kibana.yml). It happens at least if the setting 'xpack.spaces.enabled: false' is added to kibana.yml.

[1] https://github.com/elastic/kibana/blob/7.8/src/dev/build/tasks/os_packages/package_scripts/post_install.sh
[2] https://github.com/elastic/kibana/blob/7.9/src/dev/build/tasks/os_packages/package_scripts/post_install.sh
[3]

kibana/src/dev/build/tasks/os_packages/package_scripts/post_install.sh

Line 37 in 4187578

chown -R <%= user %>:<%= group %> <%= optimizeDir %>

Steps to reproduce:

1. install kibana 7.9.1 RPM downloaded from https://www.elastic.co/downloads/kibana (via yum) on Centos7
2. add 'xpack.spaces.enabled: false' to /etc/kibana/kibana.yml
3. start kibana: systemctl start kibana
4. check kibana state: systemctl status kibana

Expected behavior:
kibana should start successfully

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[mlaterman]

I've also encountered the same issue on Amazon Linux 2; running sudo chown kibana:kibana -R /usr/share/kibana/optimize/ fixed it for me.

This issue seems to be related: #6730

However I'm getting a different (maybe related?) issue when trying the same on Aliyun Linux. It fails even if i chown the directory. The output log shows:

Sep 16 23:39:37 iZuf68oiu762o1azs8nt6nZ kibana[1884]: {"type":"log","@timestamp":"2020-09-16T23:39:37Z","tags":["fatal","root"],"pid":1884,"message":"{ [Error: EACCES: permission denied, mkdir] errno: -13, code: 'EACCES', syscall: 'mkdir' }"}
Sep 16 23:39:37 iZuf68oiu762o1azs8nt6nZ kibana[1884]: {"type":"log","@timestamp":"2020-09-16T23:39:37Z","tags":["info","plugins-system"],"pid":1884,"message":"Stopping all plugins."}
Sep 16 23:39:37 iZuf68oiu762o1azs8nt6nZ kibana[1884]: FATAL  Error: EACCES: permission denied, mkdir

And I'm not certain how to debug this further

[mlaterman]

As a followup to my issues on Aliyun, I've edited the service file to run kibana through strace (/usr/bin/strace -f -e mkdir /usr/share/kibana/bin/kibana), and found that kibana is getting the access denied when trying to create the bundles subdirectory:

Sep 17 16:34:17 iZuf68oiu762o1azs8nt6nZ strace[17652]: {"type":"log","@timestamp":"2020-09-17T16:34:17Z","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":17654,"message":"Starting monitoring stats collection"}
Sep 17 16:34:18 iZuf68oiu762o1azs8nt6nZ strace[17652]: [pid 17664] mkdir("/usr/share/kibana/optimize/bundles", 0777) = -1 EACCES (Permission denied)
Sep 17 16:34:18 iZuf68oiu762o1azs8nt6nZ strace[17652]: {"type":"log","@timestamp":"2020-09-17T16:34:18Z","tags":["fatal","root"],"pid":17654,"message":"{ [Error: EACCES: permission denied, mkdir] errno: -13, code: 'EACCES', syscall: 'mkdir' }"}
Sep 17 16:34:18 iZuf68oiu762o1azs8nt6nZ strace[17652]: {"type":"log","@timestamp":"2020-09-17T16:34:18Z","tags":["info","plugins-system"],"pid":17654,"message":"Stopping all plugins."}

However, this directory already exists and is owned by the kibana user:

[root@iZuf68oiu762o1azs8nt6nZ ~]# ls -la /usr/share/kibana/optimize/
total 16
drw-r--r--  3 kibana kibana 4096 Sep 16 20:54 .
drwxr-xr-x 11 kibana kibana 4096 Sep 16 20:54 ..
-rw-r--r--  1 kibana kibana    2 Sep  1 22:37 .babel_register_cache.json
drw-r--r--  2 kibana kibana 4096 Sep 16 20:54 bundles

Edit: If I set the optimize directory permissions to 0777 it was able to start correctly

[meierfra-ergon]

@mlaterman In my case, after installing the 7.9.1 RPM, these directories (all in /usr/share/kibana/) are owned by root and therefore not writable for the kibana user:

$ docker run --rm -it centos:7
[root@58df8e1a880c /]# yum -y install wget
...
[root@58df8e1a880c /]# wget https://artifacts.elastic.co/downloads/kibana/kibana-7.9.1-x86_64.rpm
...
[root@58df8e1a880c /]# yum -y install ./kibana-7.9.1-x86_64.rpm
...
[root@58df8e1a880c /]# ls -l /usr/share/kibana/
total 1868
-rw-rw-r--    1 root root   13675 Sep  1 22:37 LICENSE.txt
-rw-rw-r--    1 root root 1826191 Sep  1 22:37 NOTICE.txt
-rw-rw-r--    1 root root    4057 Sep  1 22:37 README.txt
drwxr-xr-x    2 root root      64 Sep 18 08:04 bin
drwxr-xr-x    5 root root      43 Sep 18 08:04 built_assets
drwxr-xr-x    6 root root     108 Sep 18 08:04 node
drwxr-xr-x 1327 root root   36864 Sep 18 08:04 node_modules
drwxr-xr-x    3 root root      55 Sep 18 08:04 optimize
-rw-rw-r--    1 root root     738 Sep  1 22:37 package.json
drwxrwxr-x    2 root root       6 Sep  1 22:37 plugins
drwxr-xr-x   11 root root     160 Sep 18 08:04 src
drwxr-xr-x    2 root root      96 Sep 18 08:04 webpackShims
drwxr-xr-x    5 root root     146 Sep 18 08:04 x-pack
[root@58df8e1a880c /]# ls -l /usr/share/kibana/optimize/
total 4
drwxr-xr-x 2 root root 4096 Sep 18 08:04 bundles

If I make the same test with 7.8.1 RPM it looks like this:

[root@aa5d1d97fddb /]# ls -l /usr/share/kibana/
total 1848
-rw-rw-r--    1 root   root     13675 Jul 21 18:04 LICENSE.txt
-rw-rw-r--    1 root   root   1792297 Jul 21 18:04 NOTICE.txt
-rw-rw-r--    1 root   root      4057 Jul 21 18:04 README.txt
drwxr-xr-x    2 root   root        64 Sep 18 07:56 bin
drwxr-xr-x    5 root   root        43 Sep 18 07:56 built_assets
drwxr-xr-x    6 root   root       108 Sep 18 07:56 node
drwxr-xr-x 1641 root   root     49152 Sep 18 07:56 node_modules
drwxr-xr-x    3 kibana kibana      55 Sep 18 07:57 optimize
-rw-rw-r--    1 root   root       738 Jul 21 18:04 package.json
drwxrwxr-x    2 kibana kibana       6 Jul 21 18:04 plugins
drwxr-xr-x   11 root   root       160 Sep 18 07:57 src
drwxr-xr-x    2 root   root       114 Sep 18 07:57 webpackShims
drwxr-xr-x    5 root   root       129 Sep 18 07:57 x-pack

I`m not sure in which direction your investigation likes to go. but its quite obvious that in 7.8 rpm the post_install.sh script had chmods for this directories as pointed out in the description and the 7.9 rpm does not. So I suggest, just fix the post_install.sh and we should be good. no?

[mlaterman]

@meierfra-ergon, it seems like a fair place to start.
However I'm perplexed that changing the ownership was enough for Amazon Linux, but not for Aliyun.

[meierfra-ergon]

@mlaterman yes that is strange. It looks like that with Aliyun the the execute (x) permissions of the bundles directory was not set:
drw-r--r-- 2 kibana kibana 4096 Sep 16 20:54 bundles
this could be due to a different umask setting of this distribution.

[trackanstian]

Can confirm that we are seeing the same problem between version 7.8.x and 7.9.x

On 7.8.x the premissions are (working):

   0 drwxr-xr-x.   11 root   root       240 Sep 30 21:14 .
   4 drwxr-xr-x.   93 root   root      4096 Sep 30 21:13 ..
   0 drwxr-xr-x.    2 root   root        64 Sep 30 21:13 bin
   0 drwxr-xr-x.    5 root   root        43 Sep 30 21:13 built_assets
   4 -rw-r--r--.    1 root   root      2759 Jun 14 22:37 .i18nrc.json
  16 -rw-rw-r--.    1 root   root     13675 Jun 14 22:37 LICENSE.txt
   0 drwxr-xr-x.    6 root   root       108 Sep 30 21:13 node
  72 drwxr-xr-x. 1634 root   root     49152 Sep 30 21:13 node_modules
1756 -rw-rw-r--.    1 root   root   1794118 Jun 14 22:37 NOTICE.txt
   0 drwxr-xr-x.    3 kibana kibana      55 Sep 30 21:14 optimize
   4 -rw-rw-r--.    1 root   root       738 Jun 14 22:37 package.json
   0 drwxrwxr-x.    2 kibana kibana       6 Jun 14 22:37 plugins
   4 -rw-rw-r--.    1 root   root      4057 Jun 14 22:37 README.txt
   0 drwxr-xr-x.   11 root   root       160 Sep 30 21:14 src
   0 drwxr-xr-x.    2 root   root       114 Sep 30 21:14 webpackShims
   0 drwxr-xr-x.    5 root   root       129 Sep 30 21:14 x-pack

On 7.9.x the permissions are (this is for both 7.9.1 and 7.9.2):

   0 drwxr-xr-x.   11 root root     240 Sep 30 20:29 .
   4 drwxr-xr-x.   93 root root    4096 Sep 30 20:29 ..
   0 drwxr-xr-x.    2 root root      64 Sep 30 20:29 bin
   0 drwxr-xr-x.    5 root root      43 Sep 30 20:29 built_assets
   4 -rw-r--r--.    1 root root    2735 Sep 23 04:07 .i18nrc.json
  16 -rw-rw-r--.    1 root root   13675 Sep 23 04:07 LICENSE.txt
   0 drwxr-xr-x.    6 root root     108 Sep 30 20:29 node
  60 drwxr-xr-x. 1327 root root   36864 Sep 30 20:29 node_modules
1768 -rw-rw-r--.    1 root root 1808112 Sep 23 04:07 NOTICE.txt
   0 drwxr-xr-x.    3 root root      55 Sep 30 20:29 optimize
   4 -rw-rw-r--.    1 root root     738 Sep 23 04:07 package.json
   0 drwxrwxr-x.    2 root root       6 Sep 23 04:07 plugins
   4 -rw-rw-r--.    1 root root    4057 Sep 23 04:07 README.txt
   0 drwxr-xr-x.   11 root root     160 Sep 30 20:29 src
   0 drwxr-xr-x.    2 root root      96 Sep 30 20:29 webpackShims
   0 drwxr-xr-x.    5 root root     146 Sep 30 20:30 x-pack

We are running the keystore commands as the kibana user and this fails with permission error on 7.9.x.

Currently running: Red Hat Enterprise Linux Server release 7.8 (Maipo)

[meierfra-ergon]

Meanwhile I found out, that the described EACCES error does not occur when kibana is installed and used without any configuration changes in /etc/kibana.yml.

I forgot, that we disable some xpack features, for example we set 'xpack.spaces.enabled: false' in kibana.yml. With this setting the mentioned EACCES error happens (-> Issue description updated).

I guess, that there are cases when kibana needs to 'reorganize' something in the 'optimize' directory after config changes. If this is a valid process, the directory should be writable by the kibana user. Otherwise this process should maybe be optional? does anyone know why this has to happen at all?

[jbudz]

Have you had a chance to install 7.10? We made a sweep on all the permissions and they should be resolved with respects to the optimize directory.

For the keystore issues, we have a fix that should follow in 7.11.

[meierfra-ergon]

@jbudz I retestet with 7.10 and can now confirm, that Kibana is starting properly with our settings. And the directories show the correct ownership now.

[jbudz]

That's great to hear, thanks for the feedback. I'm going to close this out for now then, but I'll stay subscribed and we can reopen if there's any further issues.