Expand troubleshooting docs, add ESM, IAM, S3 commands (#3498)

docs/Troubleshooting.md

@@ -21,19 +21,19 @@ check "Repair S3 notifications".
 
 1. Wait a few minutes while bucket statistics and packages repopulate
 
-### Diagnose issues with ElasticSearch
+### Inspect Elasticsearch domain
 
 1. Go to CloudFormation > Stacks > YourQuiltStack > Resources
 1. Search for "domain"
 1. Click on the link for "Search" under "Physical ID"
-1. You are now under ElasticSearch > Dashboards
+1. You are now under Elasticsearch > Dashboards
 1. Set the time range to include the period before and after when you noticed
 any issues
 1. Screenshot the dashboard stats for your domain
 1. Click into your domain and then navigate to "Cluster health"
 1. Screenshot Summary, Overall Health, and Key Performance Indicator sections
 1. Send screenshots to [Quilt support](mailto:[email protected]).
-1. It is not recommended that you adjust ElasticSearch via Edit domain, as these
+1. It is not recommended that you adjust Elasticsearch via Edit domain, as these
 changes will be lost the next time that you update Quilt
 
 ## Missing metadata when working with Quilt packages via the API
@@ -139,13 +139,13 @@ and is a JSON object generated by the Quilt build system.
 
     `$ aws cloudformation describe-stack-events --stack-name QUILT_STACK_NAME`
 
-## Collect logs to report a bug
+## Collect logs to diagnose
 
 To expedite the resolution of any errors encountered while using
 Quilt, please capture the following logs and share them with
 Quilt support:
 
-### Browser Network and Console logs
+### Browser Network and Console
 
 1. Go to the affected page in your Quilt Catalog.
 1. Open the browser Developer tools:
@@ -169,21 +169,70 @@ Quilt support:
         network requests and select **Save as...**.
     1. Save the log file to your localhost.
 
-### Elastic Container Service (ECS) logs
+### Elastic Container Service (ECS)
 
 1. Find the name of your Quilt stack from querying all deployed
 stacks (in your default region, which is listed in your
 `~/.aws/config` file):
     <!--pytest.mark.skip-->
-    ```bash
+    ```sh
     aws cloudformation list-stacks
     ```
-1. Run the `aws logs` command to filter all Quilt ECS containers log entries for
-the last 30 minutes:
+1. Capture Quilt log events for the last 30 minutes as follows:
     <!--pytest.mark.skip-->
-    ```bash
+    ```sh
+    STACK_NAME="YOUR_QUILT_STACK"
     aws logs filter-log-events \
-        --log-group-name YOUR_QUILT_STACK_NAME \
+        --log-group-name "$STACK_NAME" \
         --start-time "$(( ($(date +%s) - 1800) * 1000 ))" \
         --end-time "$(( $(date +%s) * 1000 ))" > log-quilt-ecs-events.json
     ```
+
+### IAM permissions
+
+Determine which principal you're using as follows:
+<!--pytest.mark.skip-->
+```sh
+aws sts get-caller-identity
+```
+
+### S3 objects
+
+Inspect problematic objects with the following commands:
+<!--pytest.mark.skip-->
+```sh
+BUCKET="YOUR_BUCKET"
+PREFIX="YOUR_PREFIX"
+aws s3api list-object-versions --bucket "$BUCKET" --prefix "$PREFIX"
+aws s3api get-object-tagging --bucket "$BUCKET" --key "$PREFIX"
+```
+
+### Event source mapping
+
+The event source mapping is a Lambda resource that reads from SQS.
+<!--pytest.mark.skip-->
+```sh
+STACK_NAME="YOUR_QUILT_STACK"
+aws lambda get-event-source-mapping --uuid \
+    $(aws cloudformation describe-stack-resource \
+        --stack-name "$STACK_NAME" \
+        --logical-resource-id LambdaFunctionEventSourceMapping \
+        --query StackResourceDetail.PhysicalResourceId --output text)
+```
+
+## Remediation
+
+### Event source mapping
+If for some reason the event source mapping is disabled, it can be enabled as
+follows.
+<!--pytest.mark.skip-->
+```sh
+STACK_NAME="YOUR_QUILT_STACK"
+aws lambda update-event-source-mapping --uuid \
+    $(aws cloudformation describe-stack-resource \
+        --stack-name "$STACK_NAME" \
+        --logical-resource-id LambdaFunctionEventSourceMapping \
+        --query StackResourceDetail.PhysicalResourceId \
+        --output text) \
+    --enabled
+```