research: Load and Stress Testing for Grid-Proxy

[xmonader]

The goal is to perform stress and load testing (give the priority to the load testing) on the grid-proxy. Stress and load testing are crucial to ensure the service can handle high traffic and maintain optimal performance under different scenarios.

Goals

1. Identify performance bottlenecks under stress and load conditions (load is the more important one).
2. Ensure the service can handle a high volume of concurrent requests.
3. Validate the service's stability and responsiveness during peak usage.

Test Scenarios

1. Scenario 1: Peak Load

   • Simulate a scenario where the service experiences peak load to evaluate its behavior under stress.

2. Scenario 2: Concurrent Connections

   • Test the service with a large number of concurrent connections to assess its ability to handle simultaneous requests.

3. Scenario 3: Resource Utilization

   • Monitor and analyze resource utilization (CPU, memory) during stress tests to identify potential bottlenecks.

4. ... xD

Tooling Suggestions

To achieve the goals outlined above, the following tooling is recommended:

1. Load Testing Tool:

   • Consider using Vegeta or Locust for simulating realistic HTTP loads.

2. Monitoring and Profiling:

   • Integrate Prometheus for monitoring metrics such as response times, error rates, and resource utilization.
   • Utilize pprof for profiling the Go application to identify performance bottlenecks.

3. Automation:

   • Leverage continuous integration (CI) pipelines to automate the stress and load testing process.

Note: make sure to be hitting expensive endpoints too

[SalmaElsoly]

17-11-2024

created script which used locally using vegeta pkg
testing main endpoints with Prometheus metrics for:

• latency
• total success request
• success rate
• error rate
• count of http request based on status code

TODO:

• add resource utilization metrics to promtheus
• modify script
• utilizing pprof

[SalmaElsoly]

Update:

Prometheus --> resource utilization already each node has node exporter could be monitored from Grafana since tests can be running on different machine

pprof --> need to modify in the server, not compatible with the mux server we are using, can be added in the upcoming refactoring of the api