Deprecate index audit output type (#37671)

This PR deprecates the index audit output.
In general, the problem with the index audit output is that event indexing
can be slower than the rate with which audit events are generated,
especially during the daily rollovers or the rolling cluster upgrades.
In this situation audit events will be lost which is a terrible failure situation
for an audit system.
Besides of the settings under the `xpack.security.audit.index` namespace, the `xpack.security.audit.outputs` setting has also been deprecated and will be
removed in 7. Although explicitly configuring the logfile output does not touch
any deprecation bits, this setting is made redundant in 7 so this PR deprecates
it as well.

Relates #29881

docs/reference/settings/audit-settings.asciidoc

@@ -19,9 +19,14 @@ Set to `true` to enable auditing on the node. The default value is `false`.
 Specifies where audit logs are output. For example: `[ index, logfile ]`. The
 default value is `logfile`, which puts the auditing events in a dedicated
 file named `<clustername>_audit.log` on each node.
++
 You can also specify `index`, which puts the auditing events in an {es} index
 that is prefixed with `.security_audit_log`. The index can reside on the same
-cluster or a separate cluster.
+cluster or a separate cluster. deprecated[6.7.0, The outputs setting will be
+removed in 7.0 as there will only be one supported output type (`logfile`).
+Users who wish to store their audit information in an Elasticsearch index
+should write to the log file output, and a use a file ingestion component to
+index it into Elasticsearch.]
 +
 For backwards compatibility reasons, if you use the logfile output type, a
 `<clustername>_access.log` file is also created. It contains the same
@@ -34,6 +39,8 @@ For more information, see <<configuring-logging-levels>>.
 TIP: If the index is unavailable, it is possible for auditing events to
 be lost. The `index` output type should therefore be used in conjunction with
 the `logfile` output type and the latter should be the official record of events.
+This unreliability is an important reason for why the `index` output type was
+deprecated in 6.7.0 and will be removed in 7.0.
 
 --
 
@@ -116,37 +123,39 @@ these values. If the event concerns several indices, some of which are
 
 [[index-audit-settings]]
 ==== Audit Log Indexing Configuration Settings
+deprecated[6.7.0, `xpack.security.audit.index` settings namespace refers to the
+`index` audit output type which is deprecated and will be removed in 7.0]
 
 `xpack.security.audit.index.bulk_size`::
 Controls how many audit events are batched into a single write. The default
-value is `1000`.
+value is `1000`. deprecated[6.7.0]
 
 `xpack.security.audit.index.flush_interval`::
 Controls how often buffered events are flushed to the index. The default value
-is `1s`.
+is `1s`. deprecated[6.7.0]
 
 `xpack.security.audit.index.rollover`::
 Controls how often to roll over to a new index: `hourly`, `daily`, `weekly`, or
-`monthly`. The default value is `daily`.
+`monthly`. The default value is `daily`. deprecated[6.7.0]
 
 `xpack.security.audit.index.events.include`::
 Specifies the audit events to be indexed. The default value is
 `anonymous_access_denied, authentication_failed, realm_authentication_failed, access_granted, access_denied, tampered_request, connection_granted, connection_denied, run_as_granted, run_as_denied`.
 See {xpack-ref}/audit-event-types.html[Audit Entry Types] for the
-complete list.
+complete list. deprecated[6.7.0]
 
 `xpack.security.audit.index.events.exclude`::
 Excludes the specified auditing events from indexing. By default, no events are
-excluded.
+excluded. deprecated[6.7.0]
 
 `xpack.security.audit.index.events.emit_request_body`::
 Specifies whether to include the request body from REST requests on certain
-event types such as `authentication_failed`. The default value is `false`.
+event types such as `authentication_failed`. The default value is `false`. deprecated[6.7.0]
 
 `xpack.security.audit.index.settings`::
 Specifies settings for the indices that the events are stored in. For example,
 the following configuration sets the number of shards and replicas to 1 for the
-audit indices:
+audit indices: deprecated[6.7.0]
 +
 --
 [source,yaml]
@@ -169,37 +178,39 @@ even if they are unspecified (i.e. left to defaults).
 
 [[remote-audit-settings]]
 ==== Remote Audit Log Indexing Configuration Settings
+deprecated[6.7.0, `xpack.security.audit.index` settings namespace refers to the
+`index` audit output type which is deprecated and will be removed in 7.0]
 
 To index audit events to a remote {es} cluster, you configure the following
 `xpack.security.audit.index.client` settings:
 
 `xpack.security.audit.index.client.hosts`::
 Specifies a comma-separated list of `host:port` pairs. These hosts should be
-nodes in the remote cluster. If you are using default values for the 
+nodes in the remote cluster. If you are using default values for the
 <<common-network-settings,`transport.port`>> setting, you can omit the
-`port` value. Otherwise, it must match the `transport.port` setting.
+`port` value. Otherwise, it must match the `transport.port` setting. deprecated[6.7.0]
 
 `xpack.security.audit.index.client.cluster.name`::
-Specifies the name of the remote cluster.
+Specifies the name of the remote cluster. deprecated[6.7.0]
 
 `xpack.security.audit.index.client.xpack.security.user`::
 Specifies the `username:password` pair that is used to authenticate with the
-remote cluster. This user must have authority to create the `.security-audit` 
-index on the remote cluster. 
+remote cluster. This user must have authority to create the `.security-audit`
+index on the remote cluster. deprecated[6.7.0]
 
-If the remote {es} cluster has Transport Layer Security (TLS/SSL) enabled, you 
+If the remote {es} cluster has Transport Layer Security (TLS/SSL) enabled, you
 must set the following setting to `true`:
 
 `xpack.security.audit.index.client.xpack.security.transport.ssl.enabled`::
-Used to enable or disable TLS/SSL for the transport client that forwards audit 
-logs to the remote cluster. The default is `false`. 
+Used to enable or disable TLS/SSL for the transport client that forwards audit
+logs to the remote cluster. The default is `false`. deprecated[6.7.0]
 
-You must also specify the information necessary to access certificates. See 
-<<auditing-tls-ssl-settings>>. 
+You must also specify the information necessary to access certificates. See
+<<auditing-tls-ssl-settings>>.
 
 You can pass additional settings to the remote client by specifying them in the
-`xpack.security.audit.index.client` namespace. For example, you can add 
-<<modules-transport,transport settings>> and 
+`xpack.security.audit.index.client` namespace. deprecated[6.7.0] For example,
+you can add <<modules-transport,transport settings>> and
 <<tcp-settings,advanced TCP settings>> in that namespace. To allow the remote
 client to discover all of the nodes in the remote cluster you can specify the
 `client.transport.sniff` setting:

docs/reference/settings/security-settings.asciidoc

@@ -1455,7 +1455,7 @@ setting, this would be `transport.profiles.$PROFILE.xpack.security.ssl.key`.
 
 include::ssl-settings.asciidoc[]
 
-See also <<remote-audit-settings>>. 
+See also <<remote-audit-settings>>.
 
 [float]
 [[ip-filtering-settings]]

docs/reference/settings/ssl-settings.asciidoc

@@ -1,5 +1,11 @@
 
 ==== {component} TLS/SSL Settings
+ifeval::["{component}"=="Auditing"]
+deprecated[6.7.0, These settings configure the client used by the index audit
+output type which is deprecated and will be removed in 7.0. All the settings
+under the `xpack.security.audit.index` namespace are deprecated.]
+endif::[]
+
 You can configure the following TLS/SSL settings. If the settings are not configured,
 the {ref}/security-settings.html#ssl-tls-settings[Default TLS/SSL Settings]
 are used.
@@ -158,4 +164,4 @@ via the following settings:
 Set this to `PKCS11` to indicate that the PKCS#11 token should be used as a keystore.
 
 +{ssl-prefix}.truststore.type+::
-Set this to `PKCS11` to indicate that the PKCS#11 token should be used as a truststore.
+Set this to `PKCS11` to indicate that the PKCS#11 token should be used as a truststore.

x-pack/docs/en/security/auditing/auditing-search-queries.asciidoc

@@ -25,7 +25,7 @@ xpack.security.audit.logfile.events.emit_request_body: true
 ----------------------------
 --
 
-* For the `index` output:
+* For the `index` output: deprecated[6.7.0]
 +
 --
 [source,yaml]

x-pack/docs/en/security/auditing/forwarding-logs.asciidoc

@@ -1,6 +1,9 @@
 [role="xpack"]
 [[forwarding-audit-logfiles]]
 === Forwarding audit logs to a remote cluster
+deprecated[6.7.0, Forwarding audit logs is a feature of the index audit output
+type which is deprecated and will be removed in 7.0. All settings under the
+`xpack.security.audit.index` namespace are deprecated.]
 
 When you are auditing security events, you can optionally store the logs in an 
 {es} index on a remote cluster.  The logs are sent to the remote cluster by 
@@ -96,4 +99,4 @@ bin/elasticsearch-keystore add xpack.security.audit.index.client.xpack.ssl.secur
 . Restart {es}.
 
 When these steps are complete, your audit logs are stored in {es} rolling 
-indices on the remote cluster. 
+indices on the remote cluster.

x-pack/docs/en/security/auditing/output-index.asciidoc

@@ -1,6 +1,8 @@
 [role="xpack"]
 [[audit-index]]
 === Index audit output
+deprecated[6.7.0, The index output type is deprecated and will be removed in 7.0.
+The sole output for the audit trail will be the <<audit-log-output, logfile>> type.]
 
 In addition to logging to a file, you can store audit logs in Elasticsearch
 rolling indices. These indices can be either on the same cluster, or on a

x-pack/docs/en/security/auditing/output-logfile.asciidoc

@@ -55,7 +55,7 @@ The log entries in the `<clustername>_audit.log` file have the following format:
 - A field's value, a request body as well, will be escaped as per the JSON RFC 4627.
 
 There is a list of <<audit-event-types, audit event types>> specifying the
-set of fields for each sog entry type.
+set of fields for each entry type.
 
 [float]
 [[deprecated-audit-log-entry-format]]

x-pack/docs/en/security/auditing/overview.asciidoc

@@ -13,26 +13,28 @@ Audit logs are **disabled** by default. To enable this functionality, you
 must set `xpack.security.audit.enabled` to `true` in `elasticsearch.yml`.
 ============================================================================
 
-The {es} {security-features} provide two ways to persist audit logs:
+The {es} {security-features} provide two ways to persist audit logs, but only
+the first one is recommended and the other is deprecated:
 
 * The <<audit-log-output, `logfile`>> output, which persists events to
   a dedicated `<clustername>_audit.log` file on the host's file system.
   For backwards compatibility reasons, a file named `<clustername>_access.log`
   is also generated.
 * The <<audit-index, `index`>> output, which persists events to an Elasticsearch
   index. The audit index can reside on the same cluster, or a separate cluster.
+  deprecated[6.7.0]
 
 By default, only the `logfile` output is used when enabling auditing,
 implicitly outputting to both `<clustername>_audit.log` and `<clustername>_access.log`.
 To facilitate browsing and analyzing the events, you can also enable
-indexing by setting `xpack.security.audit.outputs` in `elasticsearch.yml`:
+indexing by setting `xpack.security.audit.outputs` in `elasticsearch.yml`: deprecated[6.7.0]
 
 [source,yaml]
 ----------------------------
 xpack.security.audit.outputs: [ index, logfile ]
 ----------------------------
 
 TIP: If you choose to enable the `index` output type, we strongly recommend that 
-you still use the `logfile` output as the official record of events. If the 
+you still use the `logfile` output as the official record of events. If the
 target index is unavailable (for example, during a rolling upgrade), the `index` 
-output can lose messages.
+output can lose messages. This is one reason why this output type has been deprecated.

x-pack/docs/en/security/configuring-es.asciidoc

@@ -133,7 +133,7 @@ and <<auditing-settings>>.
 
 By default, events are logged to a dedicated `elasticsearch-access.log` file in
 `ES_HOME/logs`. You can also store the events in an {es} index for
-easier analysis and control what events are logged. 
+easier analysis and control what events are logged. deprecated[6.7.0]
 --
 
 :edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/docs/reference/security/securing-communications/securing-elasticsearch.asciidoc

x-pack/plugin/deprecation/src/main/java/org/elasticsearch/xpack/deprecation/DeprecationChecks.java

@@ -50,7 +50,8 @@ private DeprecationChecks() {
             NodeDeprecationChecks::gcsRepositoryChanges,
             NodeDeprecationChecks::fileDiscoveryPluginRemoved,
             NodeDeprecationChecks::defaultSSLSettingsRemoved,
-            NodeDeprecationChecks::watcherNotificationsSecureSettingsCheck
+            NodeDeprecationChecks::watcherNotificationsSecureSettingsCheck,
+            NodeDeprecationChecks::auditIndexSettingsCheck
         ));
 
     static List<Function<IndexMetaData, DeprecationIssue>> INDEX_SETTINGS_CHECKS =

x-pack/plugin/deprecation/src/main/java/org/elasticsearch/xpack/deprecation/NodeDeprecationChecks.java

@@ -54,6 +54,22 @@ static DeprecationIssue auditLogPrefixSettingsCheck(List<NodeInfo> nodeInfos, Li
         return null;
     }
 
+    static DeprecationIssue auditIndexSettingsCheck(List<NodeInfo> nodeInfos, List<NodeStats> nodeStats) {
+        List<String> nodesFound = nodeInfos.stream()
+             .filter(nodeInfo -> (nodeInfo.getSettings().getByPrefix("xpack.security.audit.outputs").isEmpty() == false)
+                    || (nodeInfo.getSettings().getByPrefix("xpack.security.audit.index").isEmpty() == false))
+             .map(nodeInfo -> nodeInfo.getNode().getName())
+            .collect(Collectors.toList());
+        if (nodesFound.size() > 0) {
+            return new DeprecationIssue(DeprecationIssue.Level.CRITICAL,
+                "Audit index output type removed",
+                "https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes-7.0.html" +
+                    "#remove-audit-index-output",
+                "nodes with audit index output type settings: " + nodesFound);
+        }
+        return null;
+    }
+
     static DeprecationIssue indexThreadPoolCheck(List<NodeInfo> nodeInfos, List<NodeStats> nodeStats) {
         List<String> nodesFound = nodeInfos.stream()
             .filter(nodeInfo -> nodeInfo.getSettings().getByPrefix("thread_pool.index.").isEmpty() == false)

x-pack/plugin/deprecation/src/test/java/org/elasticsearch/xpack/deprecation/NodeDeprecationChecksTests.java

@@ -85,6 +85,22 @@ public void testAuditLoggingPrefixSettingsCheck() {
         assertSettingsAndIssue("xpack.security.audit.logfile.prefix.emit_node_name", Boolean.toString(randomBoolean()), expected);
     }
 
+    public void testAuditIndexSettingsCheck() {
+        DeprecationIssue expected = new DeprecationIssue(DeprecationIssue.Level.CRITICAL, "Audit index output type removed",
+                "https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes-7.0.html" + "#remove-audit-index-output",
+                "nodes with audit index output type settings: [node_check]");
+        assertSettingsAndIssue("xpack.security.audit.outputs", randomFrom("[index]", "[\"index\", \"logfile\"]"), expected);
+        assertSettingsAndIssue("xpack.security.audit.index.events.emit_request_body", Boolean.toString(randomBoolean()), expected);
+        assertSettingsAndIssue("xpack.security.audit.index.client.xpack.security.transport.ssl.enabled", Boolean.toString(randomBoolean()),
+                expected);
+        assertSettingsAndIssue("xpack.security.audit.index.client.cluster.name", randomAlphaOfLength(4), expected);
+        assertSettingsAndIssue("xpack.security.audit.index.settings.index.number_of_shards", Integer.toString(randomInt()), expected);
+        assertSettingsAndIssue("xpack.security.audit.index.events.include",
+                randomFrom("anonymous_access_denied", "authentication_failed", "realm_authentication_failed"), expected);
+        assertSettingsAndIssue("xpack.security.audit.index.events.exclude",
+                randomFrom("anonymous_access_denied", "authentication_failed", "realm_authentication_failed"), expected);
+    }
+
     public void testIndexThreadPoolCheck() {
         DeprecationIssue expected = new DeprecationIssue(DeprecationIssue.Level.CRITICAL,
             "Index thread pool removed in favor of combined write thread pool",

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/Security.java

@@ -274,7 +274,7 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw
                     s -> s.keySet().contains(SecurityField.setting("audit.outputs"))
                             ? Collections.emptyList()
                             : Collections.singletonList(LoggingAuditTrail.NAME),
-                    Property.NodeScope);
+                    Property.NodeScope, Property.Deprecated);
 
     public static final Setting<Boolean> INDICES_ADMIN_FILTERED_FIELDS_SETTING = Setting.boolSetting("indices.admin.filtered_fields", true,
             Property.NodeScope, Property.Dynamic, Property.Deprecated);
@@ -431,6 +431,7 @@ Collection<Object> createComponents(Client client, ThreadPool threadPool, Cluste
                         auditTrails.add(new DeprecatedLoggingAuditTrail(settings, clusterService, threadPool));
                         break;
                     case IndexAuditTrail.NAME:
+                        new DeprecationLogger(LOGGER).deprecated("The [index] audit type is deprecated and will be removed in 7.0");
                         indexAuditTrail.set(new IndexAuditTrail(settings, client, threadPool, clusterService));
                         auditTrails.add(indexAuditTrail.get());
                         break;

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/audit/index/IndexAuditTrail.java

@@ -127,9 +127,9 @@ public class IndexAuditTrail extends AbstractComponent implements AuditTrail, Cl
     private static final IndexNameResolver.Rollover DEFAULT_ROLLOVER = IndexNameResolver.Rollover.DAILY;
     private static final Setting<IndexNameResolver.Rollover> ROLLOVER_SETTING =
             new Setting<>(setting("audit.index.rollover"), (s) -> DEFAULT_ROLLOVER.name(),
-                    s -> IndexNameResolver.Rollover.valueOf(s.toUpperCase(Locale.ENGLISH)), Property.NodeScope);
+                    s -> IndexNameResolver.Rollover.valueOf(s.toUpperCase(Locale.ENGLISH)), Property.NodeScope, Property.Deprecated);
     private static final Setting<Integer> QUEUE_SIZE_SETTING =
-            Setting.intSetting(setting("audit.index.queue_max_size"), DEFAULT_MAX_QUEUE_SIZE, 1, Property.NodeScope);
+            Setting.intSetting(setting("audit.index.queue_max_size"), DEFAULT_MAX_QUEUE_SIZE, 1, Property.NodeScope, Property.Deprecated);
     private static final String DEFAULT_CLIENT_NAME = "security-audit-client";
 
     private static final List<String> DEFAULT_EVENT_INCLUDES = Arrays.asList(
@@ -148,22 +148,22 @@ public class IndexAuditTrail extends AbstractComponent implements AuditTrail, Cl
     private static final String FORBIDDEN_INDEX_SETTING = "index.mapper.dynamic";
 
     private static final Setting<Settings> INDEX_SETTINGS =
-            Setting.groupSetting(setting("audit.index.settings.index."), Property.NodeScope);
+            Setting.groupSetting(setting("audit.index.settings.index."), Property.NodeScope, Property.Deprecated);
     private static final Setting<List<String>> INCLUDE_EVENT_SETTINGS =
             Setting.listSetting(setting("audit.index.events.include"), DEFAULT_EVENT_INCLUDES, Function.identity(),
-                    Property.NodeScope);
+                    Property.NodeScope, Property.Deprecated);
     private static final Setting<List<String>> EXCLUDE_EVENT_SETTINGS =
             Setting.listSetting(setting("audit.index.events.exclude"), Collections.emptyList(),
-                    Function.identity(), Property.NodeScope);
+                    Function.identity(), Property.NodeScope, Property.Deprecated);
     private static final Setting<Boolean> INCLUDE_REQUEST_BODY =
-            Setting.boolSetting(setting("audit.index.events.emit_request_body"), false, Property.NodeScope);
+            Setting.boolSetting(setting("audit.index.events.emit_request_body"), false, Property.NodeScope, Property.Deprecated);
     private static final Setting<Settings> REMOTE_CLIENT_SETTINGS =
-            Setting.groupSetting(setting("audit.index.client."), Property.NodeScope);
-    private static final Setting<Integer> BULK_SIZE_SETTING =
-            Setting.intSetting(setting("audit.index.bulk_size"), DEFAULT_BULK_SIZE, 1, MAX_BULK_SIZE, Property.NodeScope);
+            Setting.groupSetting(setting("audit.index.client."), Property.NodeScope, Property.Deprecated);
+    private static final Setting<Integer> BULK_SIZE_SETTING = Setting.intSetting(setting("audit.index.bulk_size"), DEFAULT_BULK_SIZE, 1,
+            MAX_BULK_SIZE, Property.NodeScope, Property.Deprecated);
     private static final Setting<TimeValue> FLUSH_TIMEOUT_SETTING =
             Setting.timeSetting(setting("audit.index.flush_interval"), DEFAULT_FLUSH_INTERVAL,
-                    TimeValue.timeValueMillis(1L), Property.NodeScope);
+                    TimeValue.timeValueMillis(1L), Property.NodeScope, Property.Deprecated);
 
     private final AtomicReference<State> state = new AtomicReference<>(State.INITIALIZED);
     private final Settings settings;