CDPD-24496. HADOOP-17511. Add audit/telemetry logging to S3A connector (

apache#2807)

The S3A connector supports
"an auditor", a plugin which is invoked
at the start of every filesystem API call,
and whose issued "audit span" provides a context
for all REST operations against the S3 object store.

The standard auditor sets the HTTP Referrer header
on the requests with information about the API call,
such as process ID, operation name, path,
and even job ID.

If the S3 bucket is configured to log requests, this
information will be preserved there and so can be used
to analyze and troubleshoot storage IO.

Contributed by Steve Loughran.

MUST be followed by:

CDPD-28457. HADOOP-17822. fs.s3a.acl.default not working after S3A Audit feature (apache#3249)
CDPD-24982. HADOOP-17801. No error message reported when bucket doesn't exist in S3AFS

Conflicts:
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java
  hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/WriteOperationHelper.java
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AbstractStoreOperation.java
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/DeleteOperation.java
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RenameOperation.java
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/StoreContext.java
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java
	hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestPartialDeleteFailures.java

Mostly related to shaded guava.

this patch really needs CDPD-10473. HADOOP-16645. S3A Delegation Token
extension point to use StoreContext; had to CP a file in, and even then
the auditing may not be complete there. Will revisit, even though
Knox and Ranger will both need a matching change

Change-Id: Ic0a105c194342ed2d529833ecc42608e8ba2f258

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/audit/AuditConstants.java

@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.audit;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+
+/**
+ * Constants related to auditing.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+public final class AuditConstants {
+
+  private AuditConstants() {
+  }
+
+  /**
+   * The host from where requests originate: {@value}.
+   * example.org is used as the IETF require that it never resolves.
+   * This isn't always met by some mobile/consumer DNS services, but
+   * we don't worry about that. What is important is that
+   * a scan for "example.org" in the logs will exclusively find
+   * entries from this referrer.
+   */
+  public static final String REFERRER_ORIGIN_HOST = "audit.example.org";
+
+  /**
+   * Header: Command: {@value}.
+   * Set by tool runner.
+   */
+  public static final String PARAM_COMMAND = "cm";
+
+  /**
+   * Header: FileSystem ID: {@value}.
+   */
+  public static final String PARAM_FILESYSTEM_ID = "fs";
+
+  /**
+   * Header: operation ID: {@value}.
+   */
+  public static final String PARAM_ID = "id";
+
+  /**
+   * JobID query header: {@value}.
+   */
+  public static final String PARAM_JOB_ID = "ji";
+
+  /**
+   * Header: operation: {@value}.
+   * These should be from StoreStatisticNames or similar,
+   * and are expected to be at the granularity of FS
+   * API operations.
+   */
+  public static final String PARAM_OP = "op";
+
+  /**
+   * Header: first path of operation: {@value}.
+   */
+  public static final String PARAM_PATH = "p1";
+
+  /**
+   * Header: second path of operation: {@value}.
+   */
+  public static final String PARAM_PATH2 = "p2";
+
+  /**
+   * Header: Principal: {@value}.
+   */
+  public static final String PARAM_PRINCIPAL = "pr";
+
+  /**
+   * Header: Process ID: {@value}.
+   */
+  public static final String PARAM_PROCESS = "ps";
+
+  /**
+   * Thread 0: the thread which created a span {@value}.
+   */
+  public static final String PARAM_THREAD0 = "t0";
+
+  /**
+   * Thread 1: the thread making the S3 request: {@value}.
+   */
+  public static final String PARAM_THREAD1 = "t1";
+
+  /**
+   * Timestamp of span creation: {@value}.
+   */
+  public static final String PARAM_TIMESTAMP = "ts";
+
+}

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/audit/AuditStatisticNames.java

@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.audit;
+
+/**
+ * Statistic Names for Auditing.
+ */
+public final class AuditStatisticNames {
+
+  private AuditStatisticNames() {
+  }
+
+  /**
+   * Audit failure: {@value}.
+   */
+  public static final String AUDIT_FAILURE = "audit_failure";
+
+  /**
+   * A request was executed and the auditor invoked: {@value}.
+   */
+  public static final String AUDIT_REQUEST_EXECUTION
+      = "audit_request_execution";
+
+  /**
+   * Audit span created: {@value}.
+   */
+  public static final String AUDIT_SPAN_CREATION = "audit_span_creation";
+
+  /**
+   * Access check during audit rejected: {@value}.
+   */
+  public static final String AUDIT_ACCESS_CHECK_FAILURE
+      = "audit_access_check_failure";
+}