QA: v0.34 - 1/3 TM 2/3 CMT (#297)

* adding the 2/3 comet 1/3 tm case, without explanations

* moving experiment to folder to standardize names

* Description of how to run the prometheus plotter file

* regenerating figures and deleting the old ones

* moving the reporting scripts to this repo

* Updating CMT1/3 TM2/3 latency figures

* Updating prometheus metrics for cmt1 tm2 case

* Updating scripts

* extending the window to capture the long tail of experiments

* improved plotter, let's you include the average in the graph

* Updating prometheus metrics for cmt1 tm2 case

* Revert to referencing TM where it makes sense

* Updating prometheus metrics for cmt2 tm1 case

* Updating prometheus metrics for cmt2 tm1 case and references in the README.

* Updating prometheus metrics for cmt1 tm2 case

* Fixing section levels. Changed results to table

* reducing time window to 20s

* reducing time window to 20s

* Start splitting the files

* reducing time window to 20s in the script

* Update the data and comments for homogeneous network (#312)

* Changes to homogeneous results. First version

* Tweak to plotting script

* Revert "Tweak to plotting script"

* Updated homogeneous text and script

* Splitting the README.md

* Updating the instructions to plotters

* Update the latency plotter to print the experiment times in UTC and update the reference to specific experiments in the text

* Final plots for homogeneous

* Extracted and added baseline data (oct 2022) to the report

* Polishing the text

* Removed two unsued .png files

* Rename homogeneous latencies

---------

Co-authored-by: Lasaro <[email protected]>

* making file names consitent with script output and across sections. Removing files not needed. Reorganizing some sections of the report

* Plot to the imgs folder. Don't block between plots.

* Adding report for half-half configuration. Update scripts include the test.

* Fix typo in the `requirements.txt` file.
Expand the explantion of the `latency_plotter.py` script

* Moving instructions to run  to the reporting

* Small tweaks in the text

---------

Co-authored-by: Sergio Mena <[email protected]>

docs/qa/README.md

@@ -17,7 +17,8 @@ at the time the Quality Assurance process was applied for that release.
 File [method](./method.md) keeps track of the process followed to obtain the results
 used to decide if a release is passing the Quality Assurance process.
 The results obtained in each release are stored in their own directory.
-The following releases have undergone the Quality Assurance process:
+The following releases have undergone the Quality Assurance process, and the corresponding reports include detailed information on tests and comparison with the baseline.
 
-* [v0.34.x](./v034/), which was tested just before releasing v0.34.22
-* [v0.37.x](./v037/), with v.34.x acting as a baseline
+* [TM v0.34.x](./v034/TMCore.md) - Tested prior to releasing Tendermint Core v0.34.22.
+* [v0.34.x](./v034/CometBFT.md) - Tested prior to releasing v0.34.27, using TM v0.34.x results as baseline.
+* [v0.37.x](./v037/) - with TM v.34.x acting as a baseline

docs/qa/method.md

@@ -192,15 +192,24 @@ The CometBFT team should improve it at every iteration to increase the amount of
     in step 2, follow the instructions for the [`latency_throughput.py`] script.
     This plot is useful to visualize the saturation point.
 
-[`latency_throughput.py`]: ../../scripts/qa/reporting/README.md
+[`latency_throughput.py`]: ../../scripts/qa/reporting/README.md#Latency-vs-Throughput-Plotting
+
+14. Alternatively,  follow the instructions for the [`latency_plotter.py`] script.
+    This script generates a series of plots per experiment and configuration that my
+    help with visualizing Latency vs Throughput variation.
+
+[`latency_plotter.py`]: ../../scripts/qa/reporting/README.md#Latency-vs-Throughput-Plotting-version-2
 
 #### Extracting Prometheus Metrics
 
 1. Stop the prometheus server if it is running as a service (e.g. a `systemd` unit).
 2. Unzip the prometheus database retrieved from the testnet, and move it to replace the
    local prometheus database.
 3. Start the prometheus server and make sure no error logs appear at start up.
-4. Introduce the metrics you want to gather or plot.
+4. Identify the time window you want to plot in your graphs.
+5. Execute the [`prometheus_plotter.py`] script for the time window.
+
+[`prometheus_plotter.py`]: ../../scripts/qa/reporting/README.md#prometheus-metrics
 
 ## Rotating Node Testnet