Performance Benchmarking and Identification of bottlenecks

[smivij]

Before we GA the SDK we need to benchmark our performance and also improve it where it falls short of the desired performance levels. We need to identify the usage patterns for platforms, services, operations and prioritize accordingly.

---

Tasks

Preview Give feedback

1. Design doc for how Swift SDK will measure performance #1520
   +0
   
2. Implement TelemetryProvider using opentelemetry-swift (metrics) #1521
   +0
   
3. Create a swift executable that runs the top X operations #1522
   +0
   
4. Setup an OpenTelemetry Collector that runs in a docker container on ECS to emit metrics to cloudwatch
5. TBD Setup Automation of the executable
6. Create a dashbaord to display cloudwatch results

Added by @dayaffe 5/21/24

[sichanyoo]

There does not exist a cross-SDK standard for accepted benchmarking levels, because it's heavily dependent on the language. Some services to have SLA (e.g., S3Express has SLA for time it takes for operations).

To gauge which part of the SDK to benchmark, take a look at what other SDKs have done.

Also, consider making list of candidates that may take bulk of the time during request/response flow (e.g., signing for request).

Build a reusable benchmarking tool (if needed) on top of SRA observability and use that.

[sichanyoo]

Found a list of recommended metrics for benchmarking:
/BbKeALtIAOEA/Observability-Tracing-Revised#temp:C:LfW8da6814a06f242928dea883e8

[dayaffe]

See /RunBooks/PerformanceMetrics/