Document privileges required for index lifecycle management #10421

dedemorton · 2019-01-30T02:25:12Z

As a follow up to #9263, we need to document the privileges required to set up Beats to work with index life cycle management.

Note that the security docs indicate that manage_ilm is required, but the Kibana UI does not show the setting as available. I've been unable to get ILM working with security enabled and need additional input from dev.

The text was updated successfully, but these errors were encountered:

dedemorton · 2019-01-30T02:33:07Z

I was able to successfully load the ILM policy, but ran into problems when I enabled ilm in the Metricbeat config.

When I run Metricbeat with security enabled, but ILM disabled, Metricbeat ships events to Elasticsearch as expected. However, when I enable set ilm.enabled: true, I see " failed to check for alias: 403 Forbidden" errors:

2019-01-29T18:28:39.285-0800	INFO	instance/beat.go:616	Home path: [/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64] Config path: [/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64] Data path: [/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64/data] Logs path: [/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64/logs]
2019-01-29T18:28:39.287-0800	INFO	instance/beat.go:623	Beat UUID: d961ca44-ef7b-4753-a44f-f0c4626a5969
2019-01-29T18:28:39.287-0800	INFO	[beat]	instance/beat.go:936	Beat info	{"system_info": {"beat": {"path": {"config": "/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64", "data": "/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64/data", "home": "/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64", "logs": "/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64/logs"}, "type": "metricbeat", "uuid": "d961ca44-ef7b-4753-a44f-f0c4626a5969"}}}
2019-01-29T18:28:39.287-0800	INFO	[beat]	instance/beat.go:945	Build info	{"system_info": {"build": {"commit": "2c385a0764bdc537b6dc078a1d9bf11bb6d7bd95", "libbeat": "6.6.0", "time": "2019-01-24T10:38:21.000Z", "version": "6.6.0"}}}
2019-01-29T18:28:39.287-0800	INFO	[beat]	instance/beat.go:948	Go runtime info	{"system_info": {"go": {"os":"darwin","arch":"amd64","max_procs":8,"version":"go1.10.8"}}}
2019-01-29T18:28:39.288-0800	INFO	[beat]	instance/beat.go:952	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-01-22T11:08:42.135577-08:00","name":"Rhodas-MBP.hsd1.or.comcast.net","ip":["127.0.0.1/8","::1/128","fe80::1/64","fe80::143e:6265:dc42:297e/64","10.0.0.81/24","2601:1c0:7001:9df:1084:cae3:fdd0:8e21/64","2601:1c0:7001:9df:8884:ca22:5365:71ac/64","2601:1c0:7001:9df::eb2c/64","2601:1c0:7001:9df:5c01:c673:a8d3:d2d/64","fe80::c4bf:a5ff:fe61:1927/64","fe80::4b17:95f6:4b1d:8c88/64","fe80::b5f5:6b71:c879:2670/64"],"kernel_version":"16.5.0","mac":["a0:99:9b:08:ea:df","6a:00:00:67:8f:a0","6a:00:00:67:8f:a1","6a:00:00:67:8f:a0","02:99:9b:08:ea:df","c6:bf:a5:61:19:27","0a:00:27:00:00:00","0a:00:27:00:00:01"],"os":{"family":"darwin","platform":"darwin","name":"Mac OS X","version":"10.12.4","major":10,"minor":12,"patch":4,"build":"16E195"},"timezone":"PST","timezone_offset_sec":-28800,"id":"3793E7AD-D0FB-5BAB-ACFB-D6CC2B1F4AA5"}}}
2019-01-29T18:28:39.288-0800	INFO	[beat]	instance/beat.go:981	Process info	{"system_info": {"process": {"cwd": "/Users/dedemorton/BuildTesting/6.6.0_GA/metricbeat-6.6.0-darwin-x86_64", "exe": "./metricbeat", "name": "metricbeat", "pid": 39195, "ppid": 36935, "start_time": "2019-01-29T18:28:39.164-0800"}}}
2019-01-29T18:28:39.288-0800	INFO	instance/beat.go:281	Setup Beat: metricbeat; Version: 6.6.0
2019-01-29T18:28:39.288-0800	WARN	[cfgwarn]	instance/beat.go:793	BETA: Index lifecycle management is enabled which is in beta.
2019-01-29T18:28:39.288-0800	INFO	instance/beat.go:850	Set setup.template.name to 'metricbeat-6.6.0' as ILM is enabled.
2019-01-29T18:28:39.288-0800	INFO	instance/beat.go:856	Set setup.template.pattern to 'metricbeat-6.6.0-*' as ILM is enabled.
2019-01-29T18:28:39.288-0800	INFO	instance/beat.go:863	Set settings.index.lifecycle.rollover_alias in template to metricbeat-6.6.0 as ILM is enabled.
2019-01-29T18:28:39.288-0800	INFO	instance/beat.go:868	Set settings.index.lifecycle.name in template to beats-default-policy as ILM is enabled.
2019-01-29T18:28:39.288-0800	INFO	instance/beat.go:806	Set output.elasticsearch.index to 'metricbeat-6.6.0' as ILM is enabled.
2019-01-29T18:28:39.289-0800	INFO	elasticsearch/client.go:165	Elasticsearch url: http://localhost:9200
2019-01-29T18:28:39.296-0800	INFO	elasticsearch/client.go:721	Connected to Elasticsearch version 6.6.0
2019-01-29T18:28:42.329-0800	INFO	add_cloud_metadata/add_cloud_metadata.go:319	add_cloud_metadata: hosting provider type not detected.
2019-01-29T18:28:42.330-0800	INFO	elasticsearch/client.go:165	Elasticsearch url: http://localhost:9200
2019-01-29T18:28:42.331-0800	INFO	[publisher]	pipeline/module.go:110	Beat name: Rhodas-MBP.hsd1.or.comcast.net
2019-01-29T18:28:42.331-0800	INFO	instance/beat.go:403	metricbeat start running.
2019-01-29T18:28:42.331-0800	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
2019-01-29T18:28:42.333-0800	INFO	cfgfile/reload.go:150	Config reloader started
2019-01-29T18:28:42.336-0800	INFO	cfgfile/reload.go:205	Loading of config files completed.
2019-01-29T18:28:43.338-0800	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(http://localhost:9200))
2019-01-29T18:28:43.344-0800	INFO	elasticsearch/client.go:721	Connected to Elasticsearch version 6.6.0
2019-01-29T18:28:43.345-0800	ERROR	instance/ilm.go:80	Failed to check for alias: 403 Forbidden: : 
2019-01-29T18:28:45.204-0800	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: failed to check for alias: 403 Forbidden: 
2019-01-29T18:28:45.204-0800	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 1 reconnect attempt(s)
2019-01-29T18:28:45.205-0800	INFO	elasticsearch/client.go:721	Connected to Elasticsearch version 6.6.0
2019-01-29T18:28:45.211-0800	INFO	template/load.go:83	Loading template for Elasticsearch version: 6.6.0
2019-01-29T18:28:45.211-0800	INFO	template/load.go:85	Existing template will be overwritten, as overwrite is enabled.
2019-01-29T18:28:45.356-0800	INFO	template/load.go:146	Elasticsearch template with name 'metricbeat-6.6.0' loaded
2019-01-29T18:28:45.356-0800	INFO	instance/beat.go:894	Template successfully loaded.
2019-01-29T18:28:45.357-0800	ERROR	instance/ilm.go:80	Failed to check for alias: 403 Forbidden: : 
2019-01-29T18:28:47.992-0800	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: failed to check for alias: 403 Forbidden: 
2019-01-29T18:28:47.992-0800	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 2 reconnect attempt(s)
2019-01-29T18:28:47.993-0800	INFO	elasticsearch/client.go:721	Connected to Elasticsearch version 6.6.0
2019-01-29T18:28:47.997-0800	INFO	template/load.go:83	Loading template for Elasticsearch version: 6.6.0
2019-01-29T18:28:47.997-0800	INFO	template/load.go:85	Existing template will be overwritten, as overwrite is enabled.
2019-01-29T18:28:48.142-0800	INFO	template/load.go:146	Elasticsearch template with name 'metricbeat-6.6.0' loaded
2019-01-29T18:28:48.142-0800	INFO	instance/beat.go:894	Template successfully loaded.
2019-01-29T18:28:48.143-0800	ERROR	instance/ilm.go:80	Failed to check for alias: 403 Forbidden: : 
2019-01-29T18:28:52.644-0800	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: failed to check for alias: 403 Forbidden: 
2019-01-29T18:28:52.644-0800	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 3 reconnect attempt(s)
2019-01-29T18:28:52.646-0800	INFO	elasticsearch/client.go:721	Connected to Elasticsearch version 6.6.0
2019-01-29T18:28:52.651-0800	INFO	template/load.go:83	Loading template for Elasticsearch version: 6.6.0
2019-01-29T18:28:52.651-0800	INFO	template/load.go:85	Existing template will be overwritten, as overwrite is enabled.
2019-01-29T18:28:52.801-0800	INFO	template/load.go:146	Elasticsearch template with name 'metricbeat-6.6.0' loaded
2019-01-29T18:28:52.801-0800	INFO	instance/beat.go:894	Template successfully loaded.
2019-01-29T18:28:52.802-0800	ERROR	instance/ilm.go:80	Failed to check for alias: 403 Forbidden: : 
2019-01-29T18:29:02.257-0800	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: failed to check for alias: 403 Forbidden: 
2019-01-29T18:29:02.257-0800	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 4 reconnect attempt(s)
2019-01-29T18:29:02.258-0800	INFO	elasticsearch/client.go:721	Connected to Elasticsearch version 6.6.0
2019-01-29T18:29:02.262-0800	INFO	template/load.go:83	Loading template for Elasticsearch version: 6.6.0
2019-01-29T18:29:02.262-0800	INFO	template/load.go:85	Existing template will be overwritten, as overwrite is enabled.
2019-01-29T18:29:02.404-0800	INFO	template/load.go:146	Elasticsearch template with name 'metricbeat-6.6.0' loaded
2019-01-29T18:29:02.404-0800	INFO	instance/beat.go:894	Template successfully loaded.
2019-01-29T18:29:02.405-0800	ERROR	instance/ilm.go:80	Failed to check for alias: 403 Forbidden: : 
2019-01-29T18:29:12.335-0800	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":149,"time":{"ms":149}},"total":{"ticks":1016,"time":{"ms":1016},"value":1016},"user":{"ticks":867,"time":{"ms":867}}},"info":{"ephemeral_id":"0bca9f4e-8e2d-47de-9765-eabc7b4940a2","uptime":{"ms":33098}},"memstats":{"gc_next":12916816,"memory_alloc":9750840,"memory_total":419584472,"rss":43175936}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"read":{"bytes":3479},"type":"elasticsearch","write":{"bytes":355037}},"pipeline":{"clients":6,"events":{"active":83,"filtered":1,"published":83,"retry":132,"total":84}}},"metricbeat":{"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":4,"success":4},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":42,"success":42},"process":{"events":24,"success":24},"process_summary":{"events":3,"success":3},"uptime":{"events":1,"success":1}}},"system":{"cpu":{"cores":8},"load":{"1":2.4341,"15":2.5713,"5":2.6079,"norm":{"1":0.3043,"15":0.3214,"5":0.326}}}}}}
2019-01-29T18:29:21.657-0800	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: failed to check for alias: 403 Forbidden: 
2019-01-29T18:29:21.657-0800	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 5 reconnect attempt(s)
2019-01-29T18:29:21.658-0800	INFO	elasticsearch/client.go:721	Connected to Elasticsearch version 6.6.0
2019-01-29T18:29:21.664-0800	INFO	template/load.go:83	Loading template for Elasticsearch version: 6.6.0
2019-01-29T18:29:21.664-0800	INFO	template/load.go:85	Existing template will be overwritten, as overwrite is enabled.
2019-01-29T18:29:21.809-0800	INFO	template/load.go:146	Elasticsearch template with name 'metricbeat-6.6.0' loaded
2019-01-29T18:29:21.809-0800	INFO	instance/beat.go:894	Template successfully loaded.
2019-01-29T18:29:21.809-0800	ERROR	instance/ilm.go:80	Failed to check for alias: 403 Forbidden: :

dedemorton · 2019-01-30T02:36:33Z

Here's the role I created for my testing (it didn't work):

POST _xpack/security/role/metricbeat_writer
{
  "cluster": ["manage_index_templates","monitor","manage_ilm"],
  "indices": [
    {
      "names": [ "metricbeat-*" ], 
      "privileges": ["write","create_index","manage_ilm"]
    }
  ]
}

Here's my output config:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  username: "metricbeat_internal" 
  password: "MYPASSWORD"
  ilm.enabled: true
setup.template.overwrite: true

jakelandis · 2019-01-30T02:44:19Z

I suspect that you may also need "view_index_metadata" in your indices privileges. We can confirm on our side too.

dedemorton · 2019-01-30T06:53:07Z

@jakelandis Adding view_index_metadata got me a more complete message, but still errors:

	ERROR	instance/ilm.go:103	Error creating alias with write index: 403 Forbidden: {"error":
{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized 
for user [metricbeat_internal]"}],"type":"security_exception","reason":"action [indices:admin/aliases] 
is unauthorized for user [metricbeat_internal]"},"status":403},....

I tried granting the manage index privilege, and and that seemed to work (haven't verified that the rollover index gets created, but I suspect it will). Not sure manage is restrictive enough, tho.

dedemorton · 2019-01-31T00:10:09Z

Lee recommended these privileges via email:

POST _xpack/security/role/ilm
{
  "cluster": [
    // To allow creation or deleteion of policies
    "manage_ilm"
  ],
  "indices": [
    {
      "names": [
        "ilm-*",
        // needed for accessing the shrunken indices post-shrink
        "shrink-ilm-*"
      ],
      "privileges": [
        // To actually create the index (initial creation, rollover, shrink)
        "create_index",
        // needed to manage aliases for rollover
        // also for updating settings (allocation, read only, etc)
        "manage"
        // For writing to the index/alias
        "write",
        // For explain/retry/remove of policy
        "manage_ilm"]
    }
  ]
}

andrewvc · 2019-02-08T20:36:21Z

I've labeled this as a blocker because without this users will find setting up beats correctly near impossible in 7.0

andrewvc · 2019-02-08T20:40:46Z

FWIW I've had luck with

POST _xpack/security/role/heartbeat_writer
{
  "cluster": ["manage_index_templates", "monitor", "manage_ilm"],
  "indices": [
    {
      "names": [ "heartbeat-*" ], 
      "privileges": ["write","create_index"]
    }
  ]
}

I'm n ot an index rules expert though.

dedemorton · 2019-02-08T23:01:24Z

@andrewvc There's already a PR open here. Someone just need to review it. #10449

dedemorton · 2019-02-11T18:24:58Z

Closed by #10449

jakommo · 2019-04-26T16:11:17Z

I think I hit an issue with this today. It resulted in:

2019-04-26T18:03:22.337+0200	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(https://XXX)): Connection marked as failed because the onConnect callback failed: failed to check for alias 'metricbeat': (status=403) : 403 Forbidden:

I fixed it by adjusting the index name to metricbeat* rather than metricbeat-* and adding the manage privilege.
The alias is called metricbeat so the permission does not match with the dash at the end.

dedemorton · 2019-04-26T17:07:14Z

@jakommo Thanks for your input. I have a new security PR in progress that should be available for final review today. I'm pushing a bunch of changes to the PR today. I'll make sure the PR addresses your concerns. #11329

dedemorton added the docs label Jan 30, 2019

dedemorton self-assigned this Jan 30, 2019

dedemorton mentioned this issue Jan 31, 2019

Document privileges for ILM #10449

Merged

andrewvc added the blocker label Feb 8, 2019

dedemorton closed this as completed Feb 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document privileges required for index lifecycle management #10421

Document privileges required for index lifecycle management #10421

dedemorton commented Jan 30, 2019

dedemorton commented Jan 30, 2019

dedemorton commented Jan 30, 2019 •

edited

Loading

jakelandis commented Jan 30, 2019

dedemorton commented Jan 30, 2019 •

edited

Loading

dedemorton commented Jan 31, 2019

andrewvc commented Feb 8, 2019

andrewvc commented Feb 8, 2019

dedemorton commented Feb 8, 2019

dedemorton commented Feb 11, 2019

jakommo commented Apr 26, 2019

dedemorton commented Apr 26, 2019

Document privileges required for index lifecycle management #10421

Document privileges required for index lifecycle management #10421

Comments

dedemorton commented Jan 30, 2019

dedemorton commented Jan 30, 2019

dedemorton commented Jan 30, 2019 • edited Loading

jakelandis commented Jan 30, 2019

dedemorton commented Jan 30, 2019 • edited Loading

dedemorton commented Jan 31, 2019

andrewvc commented Feb 8, 2019

andrewvc commented Feb 8, 2019

dedemorton commented Feb 8, 2019

dedemorton commented Feb 11, 2019

jakommo commented Apr 26, 2019

dedemorton commented Apr 26, 2019

dedemorton commented Jan 30, 2019 •

edited

Loading

dedemorton commented Jan 30, 2019 •

edited

Loading